Week 5: The top ten developments in qualitative evaluation over the last decade – part 2

Last week Michael Quinn Patton shared the first five of his top ten trends in qualitative evaluation methods over the past decade. This week he finishes the series and identifies four challenges and opportunities he sees for qualitative approaches in the future.

5. Qualitative inquiry developments driven by evaluation practice and users’ demands

Historically, qualitative inquiry has been driven by epistemological, ontological, paradigmatic, and philosophical traditions and debates. Evaluation has brought to qualitative inquiry generally a pragmatic, utilitarian orientation. The intersection of evaluation and qualitative inquiry is being shaped by:

• Short, tight timelines​
• Large, multi-site studies
• Demands for speed

However, qualitative evaluation is only realizing a fraction of the potential contributions of in-depth qualitative field methods:

• Observation is vastly underutilized
• Interviewing dominates, and shorter interviews at that

4. Qualitative evaluation is being used intentionally as an intervention: A high degree of Process Use

Because qualitative data gathering methods and reporting approaches can be made accessible to non-academics (also known as ordinary people), and because engaging in qualitative evaluation involves learning in multiple ways at multiple levels, process use is increased through evaluation approaches that often incorporate some aspects of qualitative inquiry such as:

• Participatory or collaborative evaluation
• Feminist evaluation
• Empowerment evaluation
• Transformative evaluation (PDF)
• Developmental evaluation

These terms mean different things to different people, but share a commitment to involving the people in the setting being studied as co-inquirers, at least to some important extent, though the degree and nature of the involvement vary widely.

3. Increased value for deep contextual understanding enhances demand for and appreciation of qualitative evaluation

• The ascendance of Realist Evaluation has made contextual sensitivity paramount
• “Contextual Intelligence” is being understood as a “A Critical Competency” (Matthew R. Kutz& Anita Bamford-Wade,2013, E:CO)
• Evidence-based effective principles require contextual adaptation (as opposed to the high fidelity emphasis of best practice models)

2. The qualitative Evaluator is the Instrument: experience, expertise, and cultural competence matter

The focus in quantitative methods is on the validity and reliability of the data collection instruments and analytic procedures.  In qualitative inquiry, the experiences and capabilities of the qualitative evaluator as a person remain central to credibility:who does the work matters. In this regard, the song lyric of country music legend Waylon Jennings is germane and insightful:

Old age and treachery
Always overcomes youth and skill

1. Increased  Purposeful Sampling methods

Purposeful sampling involves selecting information-rich cases to study, cases that by their nature and substance will illuminate the evaluation question being investigated. Perhaps nothing better illustrates the difference between quantitative and qualitative methods than the different logics that support sampling approaches. The 3^rd edition (2002) of Qualitative Research and Evaluation Methods discussed 16 purposeful sampling options. The new edition presents and discusses 35 options. This development of more nuanced and targeted purposeful sampling applications captures, for me, the key to the increased utility of qualitative evaluation methods over the last decade. In the end, whatever conclusions we draw and judgments we make depend on what we have sampled. To be more strategically purposeful about sampling is to be more strategically purposeful about evaluation.

Looking Ahead: Four Challenges and Opportunities

1. Building qualitative inquiry capacity.

Global efforts are underway to strengthen evaluation capacity. Strengthening quality inquiry capacity needs to be part of that effort.

2. Increasing interest in and attracting resources to do serious, triangulated, in-depth qualitative, multi-method evaluations. 

As noted in the top ten trends, observation and in-depth fieldwork are underutilized. Interviewing and short site visits dominate.

3. Deepening evaluators’ commitment to inquire seriously into unintended consequences and take emergence (complexity) seriously.

Lip service and rhetoric give the appearance of attending to unintended consequences, but most evaluation designs devote the entire budget to assessing planned implementation and goal attainment. The kind of open-ended fieldwork needed to turn up actual consequences and emergent dynamics remains rare.

4. Cumulative-longitudinal Integration at the case and context levels

Long-term, in-depth case studies, with purposeful sampling that is sufficiently diverse to capture contextual variations, remains an ideal too rarely realized in practice.

Resources

Image: Grand Canyon Rainbow, by Jerry Miller, Flickr