RCT on a civic education program and the impact on voter behaviour by Search for Common Ground

The evaluation report by the Abdul Latif Jameel Poverty Action Lab (JPAL) is an example of how the randomized control trial (RCT) evaluation methodology can be used to test impacts of communication on behaviour and social change. In this case, the impact of civic education programs on voter and politician behavior were assessed. When compared with other civic education efforts, JPAL found evidence that SFCG’s mobile cinema increased voter knowledge of candidates, and increased policy-based (over personality based) voting intentions. This evaluation also found that these debate screenings enhanced the accountability of elected officials to their constituents following the elections. It is an example of evaluation tasks relating to investigating causal attribution and contribution, in particular: Compare results to a counterfactual (strategy 1).  A written report and a video are available.

The following information was provided to BetterEvaluation by Jessica Noske-Turner as part of the Evaluating C4D Resource Hub. 

The Evaluating C4D Resource Hub sits within BetterEvaluation and houses a growing collection of the available guides, toolkits, tools and methods to use for research monitoring and evaluation (R,M&E) of Communication for Development (C4D) initiatives. The Hub is structured around two combined frameworks:

C4D Evaluation Framework (represented by the circle) is an approach. It describes the values and principles that guide our decisions in C4D. 

The BetterEvaluation Rainbow Framework (represented by the rainbows) is a structure. It organises the practical tasks into seven categories or 'clusters' and provides options.

While the resource recommendation below discusses the resource specifically in relation to its usefulness for evaluating C4D within the Evaluating C4D Resource Hub's C4D Framework, this resource may also be of use for people working in other contexts and with different frameworks.

Authors and their affiliation

Kelly Bidwell, Katherine Casey, Rachel Glennerster

Year of publication 

2015

Type of resource

Example

Key features

One of the interesting features of this case is the way that the nature of the 'roadshow' or travelling cinema is used to create treatment and control groups for comparison (discussed on page 7-8). This is a model that could be adapted to design an RCT-style investigation for a range of communication initiatives. 

Who is this resource useful for?

• Evaluators 
• Communication Officers

How have you used or intend on using this resource?

This resource has been identified as particularly useful for evaluating media and communication initiatives, including communication for development (C4D). It could be adapted as a model for initiatives that include behaviour change objectives among both citizens and politicians where an experimental or quasi-experimental design is being considered. It was identified as part of a research project in collaboration with UNICEF C4D.

Why would you recommend it to other people?

This report would be valuable for teams considering options investigating causal attribution and contribution, and in particular those who are interested in using experimental or quasi-experimental designs to assess the impact of a program. It provides a model for making the most in differences between exposed and non-exposed groups based on the geographic locations of the 'roadshow' or travelling cinema. This example is consistent with the C4D Evaluation Framework in the following ways:  

• accountable: quasi-experimental designs provide quantitative data, which is a valued form of evidence in development agencies to show that a program works and make a case for further investment. 

On the other hand, the C4D Evaluation Framework would suggest the need to consider the following issues:

• complexity: as with all experimental and quasi-experimental designs, this creation of a counterfactual in the design of the research initiative required standardised implementation, and therefore did not allow the flexibility for adaptive and emergent approaches to C4D to be used. See other options for investigating causal attribution and contribution.
• participatory: experimental and quasi-experimental designs are generally not associated with participatory approaches, due to the need for standardisation and specific technical expertise. See other options for investigating causal attribution and contribution.