Check Intermediate Outcomes

A logic model identifies intermediate outcomes before the final impact.  If data are collected about these intermediate outcomes, and they can be linked to final impacts, it is possible to check whether all cases that achieved the final impacts achieved the intermediate outcomes.

In conjunction with options that address other tasks related to understanding causes (analyze counterfactual, and investigate exceptions) this provides a stronger causal analysis than simply reporting totals for outcomes and impacts.  

This checking can be done by crosstabulating outcomes and impacts, but only if it is possible to link outcomes and impacts data . In the example below, scenarios 1,2 and 5 could be described the same way in terms of simple percentages – 60% of families fed their children sweet potato and 60% of children increased their levels of vitamin A to clinical levels – but crosstabulations show very different patterns.

Example

Orange-colored varieties of sweet potato have been bred to increase vitamin A intake, particularly among children in Latin America.  After the introduction of these varieties to a district, and an advertising campaign to explain their value, they were sold in local markets.  The quantity of orange sweet potatoes sold was measured, as was the improvement in terms of reduced numbers of cases of Vitamin A deficiency.  The evaluation would be strengthened if it were possible to collect and analyze data to check whether those children with improved health were in families that had bought the sweet potatoes and fed them to the children.  

What is important here is both measuring intermediate outcomes (families feeding sweet potatoes to children) and also being able to link it to impacts in a crosstabulation.  This provides better information than simply being able to report, for example, that 60% of families fed the sweet potato to their children and 60% of children showed clinical improvements in their level of vitamin A.  

Scenario 1: Intermediate outcomes completely consistent with final impacts

This table shows results that are consistent with a cause-effect relationship.  All the children where families fed them sweet potato improved their vitamin A levels, and none of the other children did.

 

Children’s level of vitamin A increased to a clinically satisfactory level

Children’s level of vitamin A did not increase to this level

Total

Family fed sweet potato to children

60

0

60

Family did not feed sweet potato to children

0

40

40

Total

60

40

100

Scenario 2: Intermediate outcomes mostly consistent with final impacts

The following table shows results that are largely consistent with a cause-effect relationship.  Most of the children where families fed them sweet potato improved their vitamin A levels, and few of the other children did.  In this scenario it would be important to follow up these exceptions

 

Children’s level of vitamin A increased to a clinically satisfactory level

Children’s level of vitamin A did not increase to this level

Total

Family fed sweet potato to children

55

5

60

Family did not feed sweet potato to children

5

35

40

Total

60

40

100

Scenario 3: Intermediate outcomes not consistent with final impacts – both groups achieve impacts

The following table shows results that would not be consistent with a cause-effect relationship.  There appears no relationship.  Children not fed sweet potato were as likely to improve as those who were.  (50% of both groups improved).  In this scenario it would be important to investigate what else families were doing that might be increasing vitamin A levels outside the program.

 

Children’s level of vitamin A increased to a  clinically satisfactory level

Children’s level of vitamin A did not increase to this level

Total

Family fed sweet potato to children

30

30

60

Family did not feed sweet potato to children

20

20

40

Total

50

50

100

Scenario 4: Intermediate outcomes not consistent with final impacts – few achieve impacts despite achieving intermediate outcomes

The following table shows results that would not be consistent with a cause-effect relationship.  Even where intermediate outcomes were achieved (feeding the sweet potato), final impacts were achieved by very few.  In this scenario it would be important to use other options to explore possible explanations.
It would be important to include process evaluation – to investigate the amount of sweet potato being fed (perhaps the quantity was too low to make a clinical difference) or the way it was being fed (perhaps it was being prepared in a way that destroyed the vitamin A).
It would also be important to Investigate exceptions by gathering more detail if possible about the families which achieved the final impacts, including checking dose-response  - seeing if they were feeding more sweet potato than other families.
This type of pattern is sometimes seen when people adjust their behaviour in ways that undercut the way the intervention is supposed to work  - for example, if after medical make circumcision men engage in riskier sex than previously.  This can balance out any gains from the intervention.

 

Children’s level of vitamin A increased to a  clinically satisfactory level

Children’s level of vitamin A did not increase to this level

Total

Family fed sweet potato to children

5

55

60

Family did not feed sweet potato to children

0

40

40

Total

5

95

100

Scenario 5: Intermediate outcomes not consistent with final impacts – those achieving intermediate outcomes were less likely to achieve final impacts

The following table shows results that would not be consistent with a cause-effect relationship where the intervention contributes to producing the impacts.  In this case it shows a pattern where those engaging in the intervention have worse outcomes than those who do not.  In this scenario, children who were not fed sweet potato were more likely to improve their vitamin A levels.  This is the reverse of the results that the logic model would predict and some investigation would be needed to explore possible explanations.  

 

Children’s level of vitamin A increased

Children’s level of vitamin A did not increase

Total

Family fed sweet potato to children

20

40

60

Family did not feed sweet potato to children

40

0

40

Total

60

40

100

Advice

Advice for CHOOSING this option (tips and traps)

  • Make sure it will be possible to collect data in a way that can link outcome and impact data. This requires either collecting the data at one time or adding a personal identifier so outcome and impact data can be linked later. If you use a personal identified, ensure you develop a strategy to protect confidentiality if this has been promised.  (see Ethical evaluation)

Advice for USING this option (tips and traps)

  • Ensure processes are followed to maintain the confidentiality of data if this has been promised.  Conduct this analysis as early as possible so it is possible to follow up with the sorts of additional investigations outlined in the scenarios outlined above.
  • If you are overseeing the use of this option (manager or commissioner) ensure processes are followed to maintain the confidentiality of data if this has been promised.  Ensure this analysis is conducted as early as possible and is followed up with the sorts of additional investigations outlined in the scenarios outlined above.

Resource

Guides

Source

Funnell, S.C. and Rogers, P. J. (2011) Purposeful program theory: effective use of theories of change and logic models.  San Francisco: Jossey-Bass/Wiley. Retrievd from http://www.josseybass.com/WileyCDA/WileyTitle/productCd-0470478578.html

Updated: 14th January 2014 - 1:50am
A special thanks to this page's contributors
Contributor
Research Assistant, RMIT University.
Melbourne.

Comments

There are currently no comments. Be the first to comment on this page!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.