Broadening the range of designs and methods for impact evaluations

The working paper, written by Elliot Stern, Nicoletta Stame, John Mayne, Kim Forss, Rick Davies and Barbara Befani for the UK Department for International Development (DFID), describes how theory-based, case-based and participatory methods can be used in impact evaluations.

These designs show promise to reinforce existing IE practice, including Experimental and Statistical designs, when dealing with complex programmes.

For those with limited time, the Executive Summary provides an overview in general terms and each of the core chapters has a ‘Main Messages’ box at the beginning that highlights the key points in that chapter. The last chapter ‘Conclusions and Next Steps’ draws together 10 study conclusions.

Usefully, the report identifies not only design approaches but also variants of these approaches (or specific options). These are listed below:

Theory based

  • Causal process designs: Theory of Change, Process tracing, Contribution Analysis, Impact pathways
  • Causal mechanism designs: Realist evaluation, Congruence analysis

Case based

  • Interpretative: Naturalistic, Grounded theory, Ethnography
  • Structured: Configurations, QCA, Within-Case- Analysis, Simulations and Network analysis

Participatory

  • Normative designs: Participatory or democratic evaluation, Empowerment evaluation
  • Agency designs: Learning by doing, Policy dialogue, Collaborative, Action Research

Contents

The title of each chapter is presented below. The most important chapters, from a options perspective, are chapters 3, 4 and 5. Chapter 3 identifies and introduces the three main designs that show promise to reinforce existing IE practice when dealing with complex programmes – theory-based; case-based and participatory. How these designs are used in practice is further elaborated in Chapters 4 and 5.

  • Chapter 1: Introducing the study
  • Chapter 2: Defining Impact Evaluation
  • Chapter 3: Choosing Designs and Options 
  • Chapter 4: Evaluation Questions and Evaluation Designs
  • Chapter 5: Programme Attributes and Designs
  • Chapter 6: Quality Assurance
  • Chapter 7: Conclusions and Next Steps

Sources

Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. Department for International Development, (2012). Broadening the range of designs and options for impact evaluations (Working Paper 38). Retrieved from website: http://www.dfid.gov.uk/Documents/publications1/design-option-impact-eval.pdf

'Broadening the range of designs and methods for impact evaluations' is referenced in: