Compare results to the counterfactual
One of the three tasks involved in understanding causes is to compare the observed results to those you would expect if the intervention had not been implemented - this is known as the 'counterfactual'.
Many discussions of impact evaluation argue that it is essential to include a counterfactual. Some people however argue that in turbulent, complex situations, it can be impossible to develop an accurate estimate of what would have happened in the absence of an intervention, since this absence would have affected the situation in ways that cannot be predicted. In situations of rapid and unpredictable change, when it might not be possible to construct a credible counterfactual it might be possible to build a strong, empirical case that an intervention produced certain impacts, but not to be sure about what would have happened if the intervention had not been implemented.
For example, it might be possible to show that the development of community infrastructure for raising fish for consumption and sale was directly due to a local project, without being able to confidently state that this would not have happened in the absence of the project (perhaps through an alternative project being implemented by another organization).
For a discussion about counterfactual approaches to causal inference, see The Stanford Encyclopedia of Philosophy entry.
Methods
There are three clusters of methods for this task:
Experimental methods (or research designs)
Develop a counterfactual using a control group. Randomly assign participants to either receive the intervention or to be in a control group.
Quasi-experimental methods (or research designs)
Develop a counterfactual using a comparison group which has not been created by randomization.
Non-experimental methods
Develop a hypothetical prediction of what would have happened in the absence of the intervention.
Approaches
Expand to view all resources related to 'Compare results to the counterfactual'
Resource
- Broadening the range of designs and methods for impact evaluations
- Conditions to consider in the use of randomized experimental designs in evaluation
- High school employment: Meaningful connections for at-risk youth
- How to design and manage equity-focused evaluations
- Quasi-experimental methods for impact evaluations
- What “extras” do we get with extracurriculars? Technical research considerations
'Compare results to the counterfactual' is referenced in:
Approach
Blog
Framework/Guide
- Communication for Development (C4D) :
- Communication for Development (C4D) :
Method
Theme