EMERGENT; UNPREDICTABLE; CONTRADICTIONS; SELF-ORGANISATION
The principle of complexity draws our attention to the multiple and changing ‘interconnections’ and ‘inter-relationships’ in C4D initiatives. It highlights complicated aspects: where there are multiple organisations working in similar ways, multiple components or parts of the initiative, or where we know that C4D interventions will work differently in different contexts. It also highlights complex aspects: where change is not predictable but comes about through ‘adaptive’ responses to changing circumstances.
Where do we start?
Complexity can easily become overwhelming. It requires a different kind of mindset and can challenge our organisational systems. A good place to start is by thinking about how we need to adapt our management and organisational systems and processes to be more flexible, more attuned to different perspectives and changes. The MANAGE cluster of tasks help us reflect on whether our organisational context supports or prevents R,M&E that enables us to be adaptive in our C4D work.
Applying the C4D Principles
Manage (and commission) an evaluation or evaluation system
Understand and engage stakeholders: The stakeholders in complex social change processes may be a changing group of people. Their ideas, motivations, priorities, commitments and openness to adaptive C4D action may also change in response to the changes in the social system. Stakeholder mapping processes can help with engagement, especially where there are multiple stakeholders with different values and information needs.
Establish decision making processes: Decision-making about how the evaluation will be done (including framing its purpose and questions, choosing an evaluation team, approving an evaluation plan and an evaluation report) may need to include different stakeholders. If the key stakeholders change, the decision making structures and processes might need to be flexible. Sometimes we may need to revisit decisions that have already been made.
Develop planning documents (Evaluation Plans and M&E Frameworks): C4D is generally integrated into a program. Because of this, M&E Frameworks for C4D should ideally be developed as part of the broader program’s M&E Frameworks. Where there is a need for changing C4D action based on new insights, rapid, flexible cycles of evaluation will be most appropriate. Evaluation contracts will need to take this into account.
Develop R,M&E capacity: Capacity building efforts need to support people and organisations to become more aware of how to work with the complexity of social change. This may mean capacity building in understanding and using complexity concepts and language, and exploring different ways of thinking about and responding to social change.
Developing a program theory/logic model: A theory of change might have complicated aspects, involving multiple contributing actors, multiple goals, and different pathways linking activities to specified goals in different contexts. A theory of change might also have complex aspects able to incorporate emergent local solutions, participation by new stakeholders, introduction of new pathways and uncertain ultimate outcomes. A more detailed theory of change can be developed retrospectively using Outcome Harvesting.
Identifying potential unintended results: It is not possible to predict all the impacts that might emerge from an intervention with complex aspects. These impacts can be positive or negative, and once identified responses can be developed. Therefore R,M&E plans need to have some way of looking backwards to identify and document these (such as through open-ended questions in interviews).
Identify primary intended users: There may be different views about who to include, there may be multiple users. Primary intended users may have different information needs because of their different roles and priorities.
Deciding on the purpose: The evaluation’s purpose might need to change to support emerging findings and learning. How programs are implemented may change as a result. The primary intended users and their needs should be reviewed and revised to accommodate change.
Specify the key R,M&E questions: It it likely that there will be differing views that need to be taken into account about what the key R,M&E questions should be. In addition, the boundaries may need adjusting as situations change, particularly with the emergence of new understandings, stakeholders and ideas.
|Describe (to answer descriptive questions)||
Sample: Samples should include multiple perspectives, to understand differences in experiences in different settings. Complex interventions might need sampling strategies that can be adapted to suit emerging issues and understandings, such as using ‘purposeful’ sampling (selecting based on what is useful or most interesting) to follow up emerging patterns and findings.
Use measures, indicators or metrics: The selection and creation of outcome and impact indicators is a tricky area for C4D since emergent outcomes are hard to predict and are different in each context.
Collect and/or retrieve data: Data methods should be chosen for how well they will show different perspectives and experiences, and increase understanding of how contextual factors influence outcomes. In complicated and complex interventions, quick methods (compared to slow methods like national surveys) will be more useful for informing adaptive implementation of C4D.
Analyse data: Simple averages, frequency tables and graphs will not be enough to represent complicated and complex aspects of C4D interventions. At the very least, there should be disaggregation in tables and diagrams to show differential effects on different sub-groups. Time-lines can be important for showing non-linear change over time.
Manage data: Where there are multiple project partners, it is important to pay attention to data quality across organisations, data security when sharing data, and compatibility of IT systems. To support adaptive implementation of C4D it is useful to have data management systems that can quickly produce different types of reports in response to changing information needs.
|Understand causes (to answer questions about causes and contributions)||
Investigate Causal Attribution and Contribution: To understand the causal contribution it is important to also understand the contributions of other programs and contextual factors. Strategies to investigate this must be in the evaluation design.
Generalising findings: Although there may rarely be a one-size-fits-all set of recommendations for C4D, there may be some key principles or insights about the kinds of contextual factors that have most influence and can be generalised.
|Report and support use||
Identifying Reporting Requirements: Different primary intended users may have different preferences for receiving reports. They may also have different interests and time scales for applying the findings. Thoughtful reporting strategies that suit the user’s needs and timeframes can help facilitate an adaptive approach to C4D work.
Retrospective Analysis study of Open Defecation in Nadia District, India:
The WASH team and the C4D team in UNICEF India was working with the government on two pilot approaches to ending open defecation when a separate district (Nadia district) started gaining attention as the first district in India be declared Open Defecation Free. UNICEF India decided to undertake a study of how this had been achieved in order to see what could be learned or adapted for other parts of the country. This is consistent with the C4D Framework in the following ways:
Complexity: Targeted investigations to understand successful cases and whether aspects can be replicated and adapted elsewhere is good practice in complex situations.
Holistic: The quantitative data showed that the case was a success, and more open, holistic and qualitative methods were used to complement that knowledge to understand how and why it had worked in that case.
Critical: While the quantitative data indicated that the case was 100% successful, the study also set out to understand how different groups had been affected, and the extent to which differences in caste, wealth, geographic location, gender and other factors influence the likelihood of sustaining those changes.
Further Information and Resources
Monitoring and Evaluation of Participatory Theatre for Change - includes guidance on theories of change, recognising that multiple theories of change combine to achieve change. Although it has been developed for participatory theatre, the 'Reach, Resonance and Response' framing could be adapted to a range of C4D initiatives. Click here to read a summary and review of this resource. This resource is consistent with the C4D Evaluation Framework in the following ways:
- Complex: The guide outlines six different, interconnected theories and assumptions as part of the overall Theory of Change. It is a good example of how multiple theories can be used.
- Realistic: 'Reach, Resonance and Response' framing provides a powerful yet manageable way to think through how different theories combine in an initiative. The guide outlines six core theories of change, but encourages users to choose only those that relate to the initiative.
- Holistic: while the theories of change provided are general to participatory theatre, the guide suggests that only the relevant theories and selected, and that theories are adapted and informed by context/conflict analysis.
Democracy, Governance, and Randomised Media Assistance - (click here for a summary) draws on findings from a research report BBC Media action on the use of Randomised Control Trials (RCTs) and other experimental and quasi-experimental designs with a counterfactual in the field of media and communication for development. This review is helpful for understanding what kinds of C4D initiatives are amenable to experimental designs, and which types tend to be too complicated and complex to allow for these strategies for causal analysis.
- Accountable: program teams are often asked to consider experimental designs, since this kind of evaluation can provide credible evidence about whether a program works. However, being accountable also means understanding when this approach is feasible and will deliver credible results.
The C4D Evaluation Framework would suggest the need to reflect the following issues when considering using an experimental design:
- Complexity: as with all experimental and quasi-experimental designs, this creation of a counterfactual in the design of the research initiative required standardised implementation, and therefore did not allow the flexibility for adaptive and emergent approaches to C4D to be used.
- Participatory: experimental and quasi-experimental designs are generally not associated with participatory approaches, due to the need for standardisation and specific technical expertise.