C4D Hub: Compare results to a counterfactual (strategy 1)

One of the ways of understanding causes is to compare the observed results to those you would expect if the intervention had not been implemented. This is known as the 'counterfactual'.

There are three broad methods for creating a counterfactual. These are:

  • Experimental designs (also known as Randomised Control Trials);
  • Quasi-experimental designs (non-randomised control group);
  • Non-experimental methods for creating a counterfactual.

Experimental and quasi-experimental designs are usually used in evaluation when there is a need to prove that an intervention works, for example, in order to justify more investment or scale-up. It is less suitable as a method to explore what might work. Further, it is important to note that not all situations lend themselves to using experimental and quasi-experimental designs (discussed further below). 

General information

The BetterEvaluation Website includes comprehensive resources and overviews of the three methods (experimental designs, quasi-experimental designs, and non-experimental methods). Other key, generalist resources include:

These pages are recommended background reading before considering methods that could be applied to C4D.

Counterfactuals and C4D - Applying the C4D principles

 Complex

M&E Frameworks/Evaluations that include counterfactuals in the design are rare in C4D. Counterfactuals can be useful for explaining fairly linear cause and effect relationships, repeating patterns and interdependencies across the social system. On the other hand, the following factors make it particularly difficult:

  • Counterfactuals for evaluation generally need to be built into the design of the initiative before implementation begins. The design of the initiative will be significantly influenced by the needs of a counterfactual, especially if randomisation is used. In particular, most Counterfactual Designs require standardised implementation and are not appropriate where adaptive and emergent approaches to C4D are used.
  • Some initiatives, by their nature, are inappropriate for counterfactual designs. This is particularly the case for complicated and complex types of initiatives. 

Participatory 

Although counterfactual designs are generally not associated with participatory approaches, if the stakeholders (especially key users) decide that counterfactual designs are useful for the purpose, these groups could be involved in decision making about the design. 

Critical 

One of the strengths of a randomised control trial is that differences and inequities should become apparent through data disaggregations. However, mechanisms to create comparison groups (such as incentives) may disguise how power and marginalisation affect real-world interventions and lead to misleading results. Further, these types of designs require high levels of expertise and top-down management, which may exclude certain groups from participating in the R,M&E processes.

Critical reflection on power dynamics and inclusion might therefore suggest 

Realistic 

While experimental and quasi-experimental designs may not necessarily require more investment of time and resources, they do depend on a number of practical factors. Feasibility is dependent on: significant investment in planning and design upfront; and the ability to plan the intervention around the needs of the experimental/quasi-experimental design.  

Holistic

Experimental and quasi-experimental designs often use artificial mechanisms to create comparison groups. This might include incentives to participate, the selection of participants based on specific criteria, or additional interventions to control for other variables. These factors may distort how the intervention might work in the 'real world'. In addition, it is important even in experimental and quasi-experimental designs to undertake some additional data collection to build a holistic understanding of causes, even when the statistics appear conclusive.

C4D and Experimental designs

There are examples of Experimental Designs using Randomised Control Trials (RCTs) in C4D. Although randomization is usually done at individual participant level, it is also possible to randomise larger clusters or groups such as villages, listenership or dialogue groups, schools etc.

Resources and examples

C4D and Quasi-experimental designs

Quasi-experimental designs are in some ways more feasible since the counterfactual for comparison is created through options such as matched comparisons and double-difference designs.

Resources and examples   

C4D and Non-experimental methods

Non-experimental methods are the easiest, but also the least credible, of the three options, since it is based on developing a hypothetical prediction of what would have happened in the absence of the intervention. This can be as simple as asking key informants to predict what would have happened in the absence of the C4D initiative(s).

Expand to view all resources related to 'C4D Hub: Compare results to a counterfactual (strategy 1)'

'C4D Hub: Compare results to a counterfactual (strategy 1)' is referenced in: