Check the results are consistent with causal contribution

Methods

Gathering additional data

• Key informant attribution

  A method for testing causal reasoning by asking key informants.

• Modus operandi

  Interventions create distinctive/characteristic patterns of effects.

  Scriven describes the modus operandi as a set of footprints:

• Process tracing

  Process tracing is a case-based and theory-driven method for causal inference that applies specific types of tests to assess the strength of evidence for concluding that an intervention has contributed to changes that have been observed or measured.

Analysis

• Check dose-response patterns

  Evaluators can examine the link between dose and response as part of determining whether the program caused the outcome.

• Check intermediate outcomes

  Intermediate outcomes are identified in a logical model before the final impact. 

• Check results match a statistical model

  Program staff may develop a statistical model as part of the project theory design.

  Statistical models can be useful tools to predict elements of the program:

  Cost Time Comparison between groups
• Check results match expert predictions

  Expert predictions can be a useful part of developing the program theory.

  Program staff can draw expert predictions from the literature or by engaging a group of experts.

• Check timing of outcomes

  The program theory may predict the timing of outcomes for the evaluator to check against these dates with the dates of actual changes and outcomes.

  This is another way of checking the results support causal attribution.

• Comparative case studies

  Comparative case studies can be useful to check variation in program implementation. 

• Realist analysis of testable hypotheses

  Realist analysis of testable hypotheses tests the program theory by developing a nuanced understanding of ‘what works for whom in what circumstances and in what respects, and how?’.

• Multiple lines and levels of evidence

  Multiple lines and levels of evidence (MLLE) is a systematic approach to causal inference that involves bringing together different types of evidence (lines of evidence) and considering the strength of the evidence in terms of different indicators of a

Approaches

• These approaches combine some of the above options together with ruling out possible alternative explanations.

• Contribution analysis

  Contribution analysis is an evaluation approach that provides a systematic way of understanding an intervention's contribution to observed outcomes or impacts.

• Collaborative outcomes reporting

  Collaborative outcomes reporting (COR) is a participatory approach to impact evaluation based around a performance story that presents evidence of how a program has contributed to outcomes and impacts, that is then reviewed by both technical experts and

• RAPID outcomes assessment

  RAPID outcome assessment (ROA) is a method to assess and map the contribution of a project’s actions on a particular change in policy or the policy environment.