Impact evaluation in settings of fragility and humanitarian emergency

The paper aims to advance rigorous impact evaluations in fragile and humanitarian settings, identifying unique challenges and opportunities.

Based on interviews with researchers and funders, it offers best practices and recommendations for improving research quality, coordination, funding, and local researcher involvement.

Key content

Key takeaways from the resource include:

1. Introduction and objectives

The paper by Bakrania, Balvin, Daidone, and de Hoop (2021) aims to catalyze efforts to implement rigorous impact evaluations and empirical research in fragile and humanitarian settings. It seeks to improve the quality of evidence on interventions in these contexts, ultimately supporting effective assistance to affected populations. Drawing on interviews with researchers and funders, the paper identifies challenges, opportunities, best practices, and innovations in conducting such research.

2. Definitions and framing

The paper defines "fragile" and "humanitarian" settings, noting their overlap and unique characteristics. Fragility is understood as a combination of exposure to risks and insufficient coping capacities across economic, environmental, political, security, and societal dimensions. Humanitarian crises are defined by significant societal disruptions due to events that exceed local coping capacities. The paper also distinguishes between rigorous impact evaluations and broader empirical research, highlighting the challenges both face in these settings.

3. Literature review

The literature review synthesizes key debates on impact evaluation and research in fragile and humanitarian settings. It discusses methodological challenges, the importance of combining qualitative and quantitative methods, and the need for ethical considerations. The review underscores the limited but growing body of evidence on the effectiveness of interventions in these contexts and calls for more rigorous studies to inform decision-making.

4. Methodology

The paper's methodology includes qualitative interviews with 10 researchers and 10 research commissioners and funders. Standardized interview guides were used to structure the conversations, focusing on challenges, opportunities, best practices, and innovations. Thematic analysis of the interview transcripts was conducted by two independent researchers, with findings grouped according to the paper's objectives.

5. Analysis of interviews with researchers

• Challenges: Researchers highlighted challenges such as security risks, access to locations, and the fluid nature of emergencies. Data collection is often hindered by logistical constraints and safety concerns, necessitating flexibility and adaptability in research planning.
• Ethics: Ethical considerations are paramount, with researchers needing to balance the need for data with the potential harm to participants. Issues such as informed consent, confidentiality, and the ethical implications of findings were frequently discussed.
• Researcher skills: The importance of researchers having specific skills to navigate complex and dangerous environments was emphasized. These skills include cultural sensitivity, adaptability, and the ability to build trust with local communities.
• Funding: Funding challenges include the need for timely and flexible financial support to adapt to changing conditions. Researchers called for more streamlined funding processes that consider the unique demands of fragile and humanitarian settings.
• Relevance of research to policymakers and practitioners: Ensuring research is relevant and useful to policymakers and practitioners is crucial. Researchers stressed the importance of translating findings into actionable recommendations and engaging stakeholders throughout the research process.
• Innovations: Innovative approaches, such as using technology for data collection and analysis, were highlighted as ways to overcome traditional challenges. These innovations can improve data quality and timeliness.
• Research designs and methodologies: Researchers discussed the need for context-specific methodologies that combine qualitative and quantitative approaches. They highlighted the limitations of traditional experimental designs in these settings and the need for more flexible research strategies.

6. Analysis of interviews with research commissioners and funders

• Experiences of guiding and contracting out impact evaluations: Funders shared their experiences with commissioning impact evaluations, noting the importance of clear guidelines and the challenges of coordinating with multiple stakeholders.
• Ethics: Funders emphasized the need for ethical oversight in research projects, ensuring that evaluations do not harm participants and that ethical standards are maintained throughout the research process.
• Raising resources in fragile and humanitarian settings: Resource mobilization is a significant challenge, with funders needing to balance immediate humanitarian needs with long-term research goals. The availability of flexible and rapid funding was seen as crucial.
• Ability of donors and funders to use rigorous impact evaluations and other rigorous research in decision-making: Funders discussed how they use research findings to inform decision-making, stressing the need for evidence-based policies and programs. They also highlighted the challenges of integrating rigorous research into policy processes.
• Key research questions and priorities: Key research priorities include understanding the effectiveness of interventions, identifying best practices, and addressing the specific needs of vulnerable populations. Funders called for more research on the long-term impacts of interventions.

7. Discussion and recommendations

• Key challenges: The discussion section reiterates the key challenges identified, including security risks, ethical concerns, and funding constraints. It highlights the need for flexible and adaptive research approaches in fragile and humanitarian settings.
• Opportunities: Opportunities include leveraging technology, fostering collaboration among stakeholders, and building local research capacities. The paper emphasizes the potential for innovative approaches to improve research quality and impact.
• Innovations: Innovations discussed include the use of digital tools for data collection, new analytical methods, and collaborative research models. These innovations can help address some of the inherent challenges of conducting research in fragile and humanitarian contexts.
• Recommendations: The paper concludes with recommendations for improving the quality and quantity of impact evaluations and empirical research in fragile and humanitarian settings. These include enhancing coordination among stakeholders, providing timely and flexible funding, engaging local researchers, and ensuring ethical standards. The recommendations aim to support the delivery of high-impact assistance to people affected by conflict, natural disasters, and other humanitarian emergencies.