Check the results are consistent with causal contribution

Methods

Gathering additional data

• Asking key informants to attribute causality

  A method for testing causal reasoning by asking key informants.

• Modus operandi

  Interventions create distinctive/characteristic patterns of effects.

  Scriven describes the modus operandi as a set of footprints:

• Process tracing

  Process tracing is a case-based approach to causal inference which focuses on the use of clues within a case (causal-process observations, CPOs) to adjudicate between alternative possible explanations. 

Analysis

• Check dose-response patterns

  Evaluators can examine the link between dose and response as part of determining whether the program caused the outcome.

• Check intermediate outcomes

  Intermediate outcomes are identified in a logical model before the final impact. 

• Check results match a statistical model

  Program staff may develop a statistical model as part of the project theory design.

  Statistical models can be useful tools to predict elements of the program:

  Cost Time Comparison between groups
• Check results match expert predictions

  Expert predictions can be a useful part of developing the program theory.

  Program staff can draw expert predictions from the literature or by engaging a group of experts.

• Check timing of outcomes

  The program theory may predict the timing of outcomes for the evaluator to check against these dates with the dates of actual changes and outcomes.

  This is another way of checking the results support causal attribution.

• Comparative case studies

  Comparative case studies can be useful to check variation in program implementation. 

• Qualitative comparative analysis

  Qualitative Comparative Analysis (QCA) is a means of analysing the causal contribution of different conditions (e.g. aspects of an intervention and the wider context) to an outcome of interest.

• Realist analysis of testable hypotheses

  Realist analysis of testable hypotheses tests the program theory by developing a nuanced understanding of ‘what works for whom in what circumstances and in what respects, and how?’.

• Multiple lines and levels of evidence

  Multiple lines and levels of evidence (MLLE) is a systematic approach to causal inference that involves bringing together different types of evidence (lines of evidence) and considering the strength of the evidence in terms of different indicators of a

Approaches

• These approaches combine some of the above options together with ruling out possible alternative explanations.

• Contribution analysis

  Contribution Analysis is an approach for assessing causal questions and inferring causality in real-life program evaluations.

• Collaborative outcomes reporting

  Collaborative outcomes reporting (COR) is a participatory approach to impact evaluation based around a performance story that presents evidence of how a program has contributed to outcomes and impacts, that is then reviewed by both technical experts and

• RAPID outcomes assessment

  RAPID outcome assessment (ROA) is a method to assess and map the contribution of a project’s actions on a particular change in policy or the policy environment.