Impact evaluations and development

This guide, written by Frans Leeuw and Jos Vaessen for Network of Networks for Impact Evaluation (NONIE), provides step-by-step guidance on planning and managing an impact evaluation.

The paper includes a range of planning tools that can be used to develop an impact evaluation and a range of methods that can be used to help develop a mixed-methods approach.

Excerpt

"Interventions range along a continuum from single-“strand” initiatives with explicit objectives to complex institutional policies, and the particular type of impact evaluation would be affected by the type and scope of the intervention. Yet across this continuum, the scope of an impact evaluation can be identified through the lens of two questions: the impact of what and the impact on what?

When asking the “of what” question, it is useful to differentiate among intervention characteristics. Take single-strand initiatives with explicit objectives—for example, the change in crop yield after introduction of a new technology, or reduction in malaria prevalence after the introduction of bed nets. Such interventions can be isolated, manipulated, and measured, and experimental and quasi-experimental designs may be appropriate for assessing causal relationships between these single-strand initiatives and their effects.

At the other end of the continuum are programs with an extensive range and scope that have activities that cut across sectors, themes, and geographic areas. These can be complicated - At the other end of the continuum are programs with an extensive range and scope that have activities that cut across sectors, themes, and geographic areas. These can be complicated - At the other end of the continuum are programs with an extensive range and scope that have activities that cut across sectors, themes, and geographic areas. These can be complicated."

Contents

• Methodological and conceptual issues in impact evaluation
  • Identify the (type and scope of the) intervention
  • Agree on what is valued
  • Carefully articulate the theories linking interventions to outcomes
  • Address the attribution problem
  • Use a mixed-methods approach: The logic of the comparative advantages of methods
  • Build on existing knowledge relevant to the impact of interventions
• Managing impact evaluations
  • Determine if an impact evaluation is feasible and worth the cost
  • Start collecting data early
  • Front-end planning is important
• Appendices