Principles and strategies for overcoming challenges to evaluation in situations of conflict and fragility
The challenges of conducting evaluation in situations of conflict and fragility have been well documented. They have been widely experienced and understood and therefore are not outlined in this post.
What is less widely known and understood are the principles and strategies for overcoming these challenges, which is the focus of this post. These principles should be carried throughout the evaluation process. There is no excuse not to apply them! When applied carefully, they enhance the credibility, use and rigour of the evaluation process and its results.
The three principles and corresponding strategies are:
- Conflict analysis
- Conflict sensitivity
- Evaluating effectiveness and conflict sensitivity
1. Conflict Analysis
Conflict analysis—which includes analysis of the political economy, stakeholders, and conflict drivers and causes—is central to any evaluation of donor engagement in situations of conflict and fragility: it provides an analytical framework for assessing the relevance, effectiveness, and impact of peacebuilding activities.
Conflict analysis can be used as the basis for assessing:
- Whether activities (and the theories, strategies, etc., from which the activities are derived) have been sufficiently sensitive to the conflict setting;
- Determining the scope of the evaluation (what will be evaluated); and,
- Identifying pertinent evaluative questions.[1]
As the basis for evaluative analysis, conflict analysis is a key aspect of conflict prevention and peacebuilding evaluations regardless of the design and methods used.
Hot Resources!
- The Guidance on Conflict Analysis: Extraction of Key Points
- Conflict Sensitive approaches to development, humanitarian assistance and peace building: tools for peace and conflict impact assessment, Chapter 2 by Saferworld
- Peace and Conflict Impact Assessment Tip Sheet by SDC
- Conflict Analysis Tools by SDC
- Manual for Conflict Analysis by SIDA
2. Conflict Sensitivity
Conflict sensitivity, a key theme of this guidance, refers to the ability of an organization to:
- Understand the context in which it is operating;
- Understand the interaction between the intervention and the context; and,
- Act upon that understanding in order to avoid negative impacts and maximize positive impacts on the conflict.
All activities in fragile and conflict-affected states must be conflict sensitive. Negative impacts on a conflict may occur as a result of how a project’s outcomes contribute to peace or affect conflict, or from the operational aspects of engagement: i.e., how, where, and when donors and agencies operate and how they implement and distribute aid.
Just as a policy or intervention should be conflict sensitive, so too should the evaluation process itself. As evaluations are themselves interventions that may impact conflict, it is important to understand that questions asked as part of an evaluation may shape people’s perceptions of a conflict. Questions can be posed in ways that reinforce distrust and hostility towards ‘the other’. The operation of the evaluation may also have negative effects: the way the evaluator (and enumerators for that matter) acts, the implicit and explicit messages and values they transmit may affect the degree of risk.
The evaluation process, too, should be conflict sensitive. A number of evaluators who contributed to the guidance spoke of incidents where someone they had questioned in the course of the evaluation had been arrested or otherwise threatened. Measures to avoid this include:
- Redaction/non-inclusion of names of local members;
- Reliance upon local/national teams who are able to more easily travel in dangerous zones – though their safety must also be protected.
The potential for risk in the evaluation process should be identified at the outset of the process. Doing so is the responsibility of evaluation commissioners and team leaders, and is a requirement of conflict sensitive, ethical evaluation.
A thorough, up to date understanding of the conflict is the first step in a conflict-sensitive evaluation process. The evaluation report must explain what measures were or were not taken to ensure the conflict sensitivity of the evaluation itself and any impact that taking or not taking them may have had on the results of the evaluation.
Hot Resources!
- Conflict Sensitivity Website
- Do No Harm Handbook by Mary B. Anderson
- Fragile States Principles by OECD
- How To Guide to Conflict Sensitivity by the Conflict Sensitivity Consortium
3. Evaluating Effectiveness and Conflict Sensitivity
Being conflict sensitive and evaluating conflict sensitivity are two imperative dimensions of evaluating conflict prevention and peacebuilding work. Evaluators can help assess whether or not the standard of conflict sensitivity has been achieved – as well as provide insights on how to improve sensitivity.
Conflict sensitivity does not automatically deliver an effective peace programme or policy. A conflict sensitive intervention is not necessarily effective in addressing the key drivers of conflict and fragility. Nor are explicit peacebuilding interventions necessarily conflict sensitive.
In assessing conflict sensitivity, it is important to look at the extent to which the intervention aggravates or mitigates grievances, vulnerabilities or tensions. For interventions that do not have an explicit peacebuilding goal, evaluators would assess the effects of the development or humanitarian outputs and outcomes on the drivers of conflict and fragility, e.g., infrastructure development, a more operational police or judicial system, etc.
All activities, whether explicitly aimed at peacebuilding or not, should be examined to assess their conflict sensitivity. The Do No Harm Framework[2] identifies five ways in which operational components of an intervention may affect a conflict:
- Theft/diversion: fuelling the conflict with stolen or diverted goods/funds;
- Market effects: changing local markets with an influx of outside goods;
- Distribution: distributing goods along the lines of the conflict;
- Substitution effects: replacing existing functioning systems or structures;
- Legitimization: giving legitimacy to a group or leader by working with them.
It also identifies four ways the behavior of agencies sends messages to reinforce the modes of warfare or, alternatively, non-conflictual relations:[3]
- Conveys respect or disrespect to people and communities;
- Communicates an agency’s willingness or unwillingness to be held accountable;
- Treats people in ways that are perceived as fair or unfair;
- Demonstrates transparency or lack of transparency.
Evaluators examining conflict sensitivity, a fundamental practice of evaluation in settings of conflict and fragility, may need to examine the evaluand’s own ways of working to determine whether the intervention is conflict sensitive. This would include examining inadvertent impacts of decisions about:
- Staffing;
- Criteria for selection of beneficiaries;
- Selection of local partners;
- Relations with local authorities;
- Processes and procedures for distributing aid.[4]
Donors and agencies should not abandon their criteria or redistribute aid, but they must be aware of unintended conflict effects and develop options within the programme to mitigate them.
Hot Resources!
- Conflict Sensitivity Website
- Do No Harm Handbook by Mary B. Anderson
- How To Guide to Conflict Sensitivity by the Conflict Sensitivity Consortium
This post is taken directly from the OECD-DAC's guidance on Evaluating Peacebuilding Activities in Settings of Conflict and Fragility: Improving Learning for Results
[1] CDA, Reflecting on Peace Practice Participant Training Manual, 2009.
[2] Mary B. Anderson, Do No Harm: How Aid Can Support Peace or War, Lynne Rienner, London, 1999.
[3] Anderson, Do No Harm, 1999.
[4] Anderson, Do No Harm, 1999.