Compare results to the counterfactual

One of the three tasks involved in understanding causes is to compare the observed results to those you would expect if the intervention had not been implemented - this is known as the 'counterfactual'.

Many discussions of impact evaluation argue that it is essential to include a counterfactual.  Some people however argue that in turbulent, complex situations, it can be impossible to develop an accurate estimate of what would have happened in the absence of an intervention, since this absence would have affected the situation in ways that cannot be predicted. In situations of rapid and unpredictable change, when it might not be possible to construct a credible counterfactual it might be possible to build a strong, empirical case that an intervention produced certain impacts, but not to be sure about what would have happened if the intervention had not been implemented.

For example, it might be possible to show that the development of community infrastructure for raising fish for consumption and sale was directly due to a local project, without being able to confidently state that this would not have happened in the absence of the project (perhaps through an alternative project being implemented by another organization). 

For a discussion about counterfactual approaches to causal inference, see The Stanford Encyclopedia of Philosophy entry.

Methods

There are three clusters of methods for this task:

Experimental methods (or research designs)

Develop a counterfactual using a control group. Randomly assign participants to either receive the intervention or to be in a control group.

  • Control group

    A control group is an untreated research sample against which all other groups or samples in the research is compared.

Quasi-experimental methods (or research designs)

Develop a counterfactual using a comparison group which has not been created by randomization.

  • Difference-in-difference

    Difference-in-difference involves comparing the before-and-after difference for the group receiving the intervention (where they have not been randomly assigned) to the before-after difference for those who did not.

  • Instrumental variables

    This method is used to estimate the causal effect of variables on an intervention.

  • Judgemental matching

    Judgemental matching involves creating a comparison group by finding a match for each person or site in the treatment group based on researcher judgements about what variables are important.

  • Matched Comparisons

    When using Matched Comparisons, participants (individuals, organizations or communities) are each matched with a non-participant on variables that are thought to be relevant which can be difficult to adequately match on all relevant criteria.

  • Propensity scores

    Propensity score matching (PSM) is a quasi-experimental method used to estimate the difference in outcomes between beneficiaries and non-beneficiaries that is attributable to a particular program.

  • Regression discontinuity

    Regression Discontinuity Design (RDD) is a quasi-experimental evaluation option that measures the impact of an intervention, or treatment, by applying a treatment assignment mechanism based on a continuous eligibility index which is a variable with a co

  • Sequential allocation

    Sequential allocation involves creating a treatment group and a comparison group by using a sequence to choose participants (e.g. every 3rd person on the list).

  • Statistically created counterfactual

    A statistical model, such as regression analysis, is used to develop an estimate of what would have happened in the absence of an intervention.

Non-experimental methods

Develop a hypothetical prediction of what would have happened in the absence of the intervention.

  • Key informant

    Asking experts of programmes or in the community to predict what would have happened in the absence of the intervention.

  • Logically constructed counterfactual

    In some cases it is not possible to construct a counterfactual by creating a control group or a comparison group, but by constructing one logically.

Approaches

  • Randomised controlled trial

    Randomised controlled trials (RCTs), or randomised impact evaluations, are a type of impact evaluation that uses randomised access to social programmes as a means of limiting bias and generating an internally valid impact estimate.

Expand to view all resources related to 'Compare results to the counterfactual'

'Compare results to the counterfactual' is referenced in:

Back to top