Week 32: Better use of case studies in evaluation
Case studies are often used in evaluations – but not always in ways that use their real potential.
Recently I had an opportunity to spend some time with the evaluation unit of UNOIOS (United Nations Office of Internal Oversight and Inspection) and some of their UN evaluation colleagues exploring ways to better use case studies in evaluation. Here are five lessons I took away from our time together.
1. Be clear about what you mean by a case study
Case study is a research design that involves an intensive study of one or more cases rather than an extensive study of many, and which involve multiple sources of evidence – often a combination of quantitative and qualitative data.
Be clear about what the case is – is it a person, a site, a project, an event, a procedure, a country, or something else? And what is it a case of? A case of successful implementation - or a case that illustrates the barriers to successful implementation? A typical day? A small project, as compared to a large project?
2. Be clear about why you are doing a case study – and then choose the type of case study that matches this.
There are different types of case studies. Choose the right one for your purpose. This list draws on a guide "Case Studies in Evaluation" produced by the United State Government Accounting Office, which identified six different types of case studies – and adds one more (comparative case study):
Illustrative: This is descriptive in character and intended to add realism and in-depth examples to other information about a program or policy. They are especially useful in evaluations intended to be used by people without direct experience of a program or a situation. These are often used to complement quantitative data by providing examples of the overall findings. These can range from brief narratives to detailed, vivid, and evocative narratives that provide a vicarious experience and allow readers to understand the connections and meanings.
Exploratory: This is also descriptive but is aimed at generating hypotheses for later investigation rather than simply providing illustration. This type of case study is done before planning a component of the evaluation which will involve extensive data collection (such as a survey).
Critical instance: This examines a single instance of unique interest, or serves as a critical test of an assertion about a program, problem or strategy.
Program implementation: This investigates operations, often at several sites, and often with reference to a set of norms or standards about implementation processes.
Program effects: This examines the causal links between the program and observed effects (outputs, outcomes or impacts, depending on the timing of the evaluation) and usually involves multisite, multimethod evaluations. It involves detailed and strategic data collection to identify and test different theories about what has produced the observed impacts.
Cumulative: This brings together findings from many case studies to answer evaluative questions.
Comparative case studies: These are not only multiple case studies but ones which are designed to use the comparisons between the cases to build and test hypotheses.
3. Match sampling, data collection, analysis and reporting to the type of case
In evaluation, it is very unlikely that people will be interested in the case itself without wanting to know how to use those findings to think about a larger population. Case studies usually use some form of purposeful sampling – random sampling is rarely appropriate (unless this is the only form of sampling that will be credible to the evaluation users).
Carefully select the type of purposeful sampling so that appropriate inference or translation of findings can be made. What I often see in evaluations is inappropriate choice of sampling type which then does not match the type of inference needed. For example it would not be appropriate to sample extreme cases (such as a very successful site) and then draw conclusions as if the sample were typical cases. I often see case studies where the cases have been chosen in terms of a maximum variety sample drawn across a number of dimensions (for example, choosing countries which show a range of levels of development, region and some relevant contextual factors such as institutional arrangements in the country) - but then the evaluation is not clear about how to use this information to say something about the larger group.
Being clear about the type of case, and the type of inference that will be made, can make it clear what sort of sample is needed. For example, an illustrative case study might be done of a case which is identified as "typical" along some dimensions, in order to show what an average case is like. Or outlier sampling might be used to show what the program looks like when it works particularly well or badly. Or maximum variation sampling might be used to show the range of what it looks like in different situations.
An exploratory case study might use theory-based sampling, identifying important sub-groups according to the theory of change and sampling from each. This could be used to develop theories of change for each case and compare them to see how they differ across different cases, or to develop an overall theory of change for the whole program or for types of projects that can be used to guide the next stage of data collection.
4. Link case studies thoughtfully to other elements of an evaluation or a monitoring and evaluation system
Think carefully about when the case studies should be done and how they can be linked. For example, exploratory case studies can be useful to do before a survey; explanatory case studies are likely to be useful after a survey.
5. Create opportunities for iteration
If possible, don’t commit the entire evaluation budget at the beginning but set some aside to follow up emerging findings and test hypotheses by doing additional work such as:
- More data analysis of existing data from cases
- More data collection and analysis from existing cases
- Adding more cases
You can find more resources about using case studies in evaluation on the Case Study approach page on the BetterEvaluation site.
Do you have other good resources or examples to share? Do you have questions about using case studies in evaluation?