Introduction to mixed methods in impact evaluation

This guide, written by Michael Bamberger for InterAction outlines the elements of a mixed methods approach with particular reference to how it can be used in an impact evaluation.

Excerpt

"There is rarely a single evaluation methodology that can fully capture all of the complexities of how programs operate in the real world. Consequently, evaluators must find creative ways to combine different evaluation frameworks, tools and techniques—hence the growing interest in MM approaches. The unique feature of mixed methods approaches is that they seek to integrate social science disciplines with predominantly QUANT and predominantly QUAL approaches to theory, data collection and data analysis and interpretation.

Although many evaluators now routinely use a variety of methods, “What distinguishes mixed-method evaluation is the intentional or planned use of diverse methods for particular mixed-method purposes using particular mixed-method designs” (Greene 2005:255). Most commonly, methods of data collection are combined to make an evaluation MM, but it is also possible to combine conceptual frameworks, hypothesis development, data analysis, or frameworks for the interpretation of the evaluation findings." (Bamberger 2012)

Contents

  • Part I. Why mixed methods? 3
  • What is a mixed-methods impact evaluation design? 3
  • The limitations of an exclusive reliance on QUANT or QUAL evaluation approaches 3
  • The benefits of a mixed methods approach 4
  • Part II. The mixed methods approach 9
  • Four decisions for designing a mixed methods evaluation 9
  • Applying MM approaches at each stage of the evaluation 13
  • Part III. Application of mixed methods designs 19
  • Sampling strategies for QUANT and QUAL oriented MM evaluations 19
  • Using mixed methods to evaluate complex interventions 21
  • Assessing processes of behavioural change 24
  • Part IV. Managing mixed methods evaluations 27
  • Mixed methods designs require a special management approach 27
  • Tips for resource-constrained NGOs to mobilize the expertise and resources required to conduct mixed-methods evaluations 30
  • Part V. Case Studies Illustrating Different Applications of mixed Methods designs 32
  • Annexes (available at http://www.interaction.org/impact-evaluation-notes(archived link))
  • Annex 1. Strengths and weaknesses of quantitative evaluation designs
  • Annex 2. Strengths and weaknesses of qualitative evaluation designs
  • Annex 3. Examples of evaluation designs at each point on the QUANT - QUAL continuum
  • Annex 4. Characteristics of QUANT and QUAL approaches at different stages of the evaluation
  • Annex 5. How QUANT and QUAL approaches complement each other at different stages of an evaluation
  • Annex 6. Comparing random and purposive sampling methods
  • Annex 7. A range of quantitative, qualitative and theory-based approaches for defining the counterfactual
  • Annex 8. Strategies for reducing the costs of data collection and analysis
  • Annex 9. Example of triangulation: comparing estimates of household income and poverty from different sources
  • Annex 10. Case studies of MM evaluation designs with predominant QUANT, QUAL and balanced orientations
  • Annex 11. How mixed methods can strengthen QUANT evaluation designs

Sources

Bamberger, M. InterAction, (2012). Introduction to mixed methods in impact evaluation (No. 3.). Retrieved from website: https://www.interaction.org/wp-content/uploads/2019/03/Mixed-Methods-in-Impact-Evaluation-English.pdf

'Introduction to mixed methods in impact evaluation' is referenced in: