52 weeks of BetterEvaluation: Week 31: A series on mixed methods in evaluation

Simon Hearn's picture 27th July 2013 by Simon Hearn

This week we are focusing on mixed methods in evaluation. We'll have two further blogs on the subject, one exploring an evaluation that used mixed methods and the other asking whether we are clear enough about what mixed methods really means - there are many evaluations out there claiming to be mixed methods when all they do is supplement a qualitative survey with interview data. 

Read the part 2 and part 3 of the mini-series on mixed methods.

Mixed methods are very much in the lime-light at the moment with a recent special edition of the New Direction for Evaluation journal devoted to mixed methods and a brand new professional association formed this month - the Mixed Methods International Research Association. Indeed, no evaluation toolkit would be complete without options for combining quantitative and qualitative data

To get us going on this topic I wanted to ask: what do you understand as mixed methods in evaluation and why is it so important?

The latter question is the simpler and points to one of the principles of BetterEvaluation: Many of the questions that evaluations attempt to address require, by their nature, different kinds of data, different perspectives and different methods combined in clever ways to generate the evidence needed to make the required value judgments.

I've already pointed to my opinion that mixed methods is more than the mere use of qualitative methods alongside quantitative methods and I think this view is commonly shared. In a new paper from Michael Bamberger published by Social Impact mixed methods evaluation is defined as:

"An approach to evaluation that systematically integrates QUANT and QUAL methodologies and methods at all stages of an evaluation."

You can see that Michael emphasises the fact that quantitative and qualitative methods have to be systematically integrated (not just used one after the other) at all stages of the evaluation (not just in data collection) for a design to be called mixed method.

Howard White takes a slightly different view of mixed methods and defines them in terms of mixing counterfactual analysis with factual analysis which can be quantitative or qualitative. It's not the methods in this case but the mode of analysis that is mixed. 

Other thinkers in this field go further than this and describe mixed methods as a fundamental and deliberate attempt to reconcile two very different scientific paradigms. A more recent book provides fascinating insight into the qualitative and quantitative paradigms and how they can be resolved.

But for all the methodological wrangling, what can we as evaluators or evaluation commissioners actually do?

BetterEvaluation provides advice for a number of options that can be considered once the task of combining quantitative and qualitative data has been decided. It provides information on how to collect data (in parallel or in sequence), how to combine data (component or integrated design) and different purposes of combining data (enriching, examining, explaining and triangulation).

The second post in this series was: Mixed methods in evaluation part 2: exploring the case of a mixed-method outcome evaluation. The third wasMixed methods in evaluation Part 3: Enough pick and mix; time for some standards on mixing methods in impact evaluation.

[Image source: christmasstockimages.com]


52 Weeks of BetterEvaluation. Click here to view past features

A special thanks to this page's contributors
Research Fellow, Overseas Development Institute.


rickjdavies's picture
rick davies

Hi Simon
Glad to see you are giving "A tale of two cultures" some air time. I think it is one the most interesting and informative books on qualitative methods I have read for ages. Actually its a platform for a particular view of qualitative methods rather than an exposition about the whole spectrum of qualitative methods, but its nevertheless a very informative view, for me at least.

Goertz, Gary, and James Mahoney. 2012. A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences. Princeton University Press. http://www.amazon.co.uk/Tale-Two-Cultures-Qualitative-Quantitative/dp/06....


Simon Hearn's picture
Simon Hearn

Thanks Rick - for putting me on the book in the first place. It's in my pile of books for holiday reading.

John Cosgrave's picture
John Cosgrave

<p>An earlier version of Mahoney and Goertz is available online at: <a href="http://www.ib.ethz.ch/teaching/pwgrundlagen/Goertz_Mahoney_2006.pdf/" style="font-size: 14px; line-height: 1.2;">http://www.ib.ethz.ch/teaching/pwgrundlagen/Goertz_Mahoney_2006.pdf/</a>...
<p>This surprisingly readable (for the topic) essay contrasts the two research traditions across 10 areas: (1) approaches to explanation, (2) conceptions of causation, (3) multivariate explanations, (4) equifinality, (5) scope and causal generalization, (6) case selection, (7) weighting observations, (8) substantively important cases, (9) lack of fit, and (10) concepts and measurement.</p>
<p><span style="font-size: 14px; line-height: 1.2;">The article notes that the labels quantitative and qualitative do a poor job capturing the real differences between the traditions. Quantitative analysis inherently involves the use of numbers, but all statistical </span>analyses<span style="font-size: 14px; line-height: 1.2;"> also rely heavily on words for interpretation. Qualitative studies quite frequently employ numerical data; many qualitative techniques in fact require quantitative information. Although they have no illusions about changing prevailing terminology, the authors note that better labels for describing the two kinds of research analyzed here would be statistics versus logic, effect estimation versus outcome explanation, or population-oriented versus case-oriented approaches.</span></p>
<p><span style="font-size: 14px; line-height: 1.2;">Mahoney, J., &amp; </span>Goertz<span style="font-size: 14px; line-height: 1.2;">, G. (2006). A Tale of Two Cultures: Contrasting Quantitative and Qualitative Research. Political Analysis, 14 (3), 227-249.&nbsp;</span></p>

Simon Hearn's picture
Simon Hearn

Excellent. Thanks John. Great to have a free, publicly available copy. We'll add it as a resource to the site.

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.