C4D Hub: Compare results to a counterfactual (strategy 1)

One of the ways of understanding causes is to compare the observed results to those you would expect if the intervention had not been implemented. This is known as the 'counterfactual'.

There are three broad methods for creating a counterfactual. These are:

• Experimental designs (also known as Randomised Control Trials);
• Quasi-experimental designs (non-randomised control group);
• Non-experimental methods for creating a counterfactual.

Experimental and quasi-experimental designs are usually used in evaluation when there is a need to prove that an intervention works, for example, in order to justify more investment or scale-up. It is less suitable as a method to explore what might work. Further, it is important to note that not all situations lend themselves to using experimental and quasi-experimental designs (discussed further below). 

General information

The BetterEvaluation Website includes comprehensive resources and overviews of the three methods (experimental designs, quasi-experimental designs, and non-experimental methods). Other key, generalist resources include:

• Resources on Randomised Control Trials by UNICEF Office of Research Innocenti, including a short video 
• Resources on Quasi Experimental Design by the UNICEF Office of Research Innocenti 
• Resources and toolkits via 3ie (International Initiative for Impact Evaluation) - note that impact evaluation in this context is used as interchangeable with experimental designs. 
• A guide by JPAL (Latif Jameel Poverty Action Lab), which takes users through the steps of deciding if a question can be answered through an experimental design and randomisation, through to research designs, data collection and analysis. 

These pages are recommended background reading before considering methods that could be applied to C4D.

Counterfactuals and C4D - Applying the C4D principles

 Complex

M&E Frameworks/Evaluations that include counterfactuals in the design are rare in C4D. Counterfactuals can be useful for explaining fairly linear cause and effect relationships, repeating patterns and interdependencies across the social system. On the other hand, the following factors make it particularly difficult:

• Counterfactuals for evaluation generally need to be built into the design of the initiative before implementation begins. The design of the initiative will be significantly influenced by the needs of a counterfactual, especially if randomisation is used. In particular, most Counterfactual Designs require standardised implementation and are not appropriate where adaptive and emergent approaches to C4D are used.
• Some initiatives, by their nature, are inappropriate for counterfactual designs. This is particularly the case for complicated and complex types of initiatives. 

Participatory 

Although counterfactual designs are generally not associated with participatory approaches, if the stakeholders (especially key users) decide that counterfactual designs are useful for the purpose, these groups could be involved in decision making about the design. 

Critical 

One of the strengths of a randomised control trial is that differences and inequities should become apparent through data disaggregations. However, mechanisms to create comparison groups (such as incentives) may disguise how power and marginalisation affect real-world interventions and lead to misleading results. Further, these types of designs require high levels of expertise and top-down management, which may exclude certain groups from participating in the R,M&E processes.

Critical reflection on power dynamics and inclusion might therefore suggest 

• Strategy 2: Check the results support causal attribution and
• Strategy 3: Investigate possible alternative explanations 

Realistic 

While experimental and quasi-experimental designs may not necessarily require more investment of time and resources, they do depend on a number of practical factors. Feasibility is dependent on: significant investment in planning and design upfront; and the ability to plan the intervention around the needs of the experimental/quasi-experimental design.  

Holistic

Experimental and quasi-experimental designs often use artificial mechanisms to create comparison groups. This might include incentives to participate, the selection of participants based on specific criteria, or additional interventions to control for other variables. These factors may distort how the intervention might work in the 'real world'. In addition, it is important even in experimental and quasi-experimental designs to undertake some additional data collection to build a holistic understanding of causes, even when the statistics appear conclusive.

C4D and Experimental designs

There are examples of Experimental Designs using Randomised Control Trials (RCTs) in C4D. Although randomization is usually done at individual participant level, it is also possible to randomise larger clusters or groups such as villages, listenership or dialogue groups, schools etc.

Resources and examples

• BBC Media action has published a review of the use of RCTs and other experimental and quasi-experimental designs with a counterfactual in the field of media and communication for development. It includes examples of using radio listening groups for a comparison of exposed and counterfactual groups. See 
• Femina HIP, a Tanzanian C4D NGO, partnered with researchers to implement a RCT design of an edutainment TV program . The television program was intended to encourage entrepreneurialism among youth. The quasi-experimental design involved randomly selecting 43 secondary schools, and using encouragement design to incentivise viewing of the TV program by students at half the schools. Importantly, the researchers also conducted focus group discussions. The focus group discussions revealed that young people don't always have the power within the household to choose what they watch. This means that even though the results of the RCT showed that viewing the TV program lead to increases in entrepreneurial attitudes and behaviours, young people only had access to the TV content because of the incentives offered. This shows the importance of using different methods to understand contextual factors, even when using RCTs.  
• An RCT on a civic education program and the impact on voter behaviour by Search for Common Ground with Jpal in Sierra Leone. 
• Delaying Child Marriage through Community-Based Skills-Development Programs for Girls: Results from a Randomized Controlled Study in Rural Bangladesh assesses the impact of the Bangladeshi Association for Life Skills, Income, and Knowledge for Adolescents (BALIKA) programme by performing a difference-in-differences (DiD) analysis adjusting for three key sociodemographic characteristics: age, religion, and wealth quintile.
• Encouraging community-based monitoring of healthcare in Uganda is a case study by Jpal which shows how an RCT design can be applied to assess participatory approaches. In this case, the unit is the village. 25 village dispensaries were randomly selected to begin community monitoring processes, with 25 other dispensaries used as the control (no treatment).  

C4D and Quasi-experimental designs

Quasi-experimental designs are in some ways more feasible since the counterfactual for comparison is created through options such as matched comparisons and double-difference designs.

Resources and examples   

• BBC Media action has published a review of the use of RCTs and other experimental and quasi-experimental designs with a counterfactual in the field of media and communication for development. It includes examples of using radio listening groups for a comparison of exposed and counterfactual groups. See 

C4D and Non-experimental methods

Non-experimental methods are the easiest, but also the least credible, of the three options, since it is based on developing a hypothetical prediction of what would have happened in the absence of the intervention. This can be as simple as asking key informants to predict what would have happened in the absence of the C4D initiative(s).