Week 19: Ways of framing the difference between research and evaluation
One of the challenges of working in evaluation is that important terms (like ‘evaluation’, ‘impact’, ‘indicators’, ‘monitoring’ and so on ) are defined and used in very different ways by different people.
Sometimes the same word is used but to mean quite different things; other times different words are used to mean the same thing. And, most importantly, many people are simply unaware that other people use these words in these different ways.
Since BetterEvaluation seeks to support discussion and learning across organisational, sectoral, and disciplinary boundaries, it is important for us to find ways to understand each other.
This is the first in what we plan to be a series about key terms in evaluation. It sets out four different ways of thinking about research as compared to evaluation – and when these might be useful ways of using them.
A perennial question on discussion forums, there are surprisingly different ways of thinking about how evaluation is different to research. These can have important implications for how we plan, manage, conduct and train people to do evaluation – and communicate to others about the process and practice of evaluation.
1. Evaluation and research as a dichotomy
One of the most common ways of thinking about the difference between research and evaluation is as a dichotomy - two distinct and mutually exclusive categories. Research is seen as more interested in producing generalisable knowledge, more theoretical, more controlled by the researchers – evaluation is seen as more interested in specific, applied knowledge, and more controlled by those funding or commissioning the evaluation.
For example, the new Evaluation Flash Cards developed by Michael Quinn Patton for the Otto Bremer Foundation includes the following flashcard on “Evaluation versus Research”.
| Research | Evaluation |
|---|---|
| Purpose is testing theory and producing generalizable findings. | Purpose is to determine the effectiveness of a specific program or model. |
| Questions originate with scholars in a discipline. | Questions originate with key stakeholders and primary intended users of evaluation findings. |
| Quality and importance judged by peer review in a discipline. | Quality and importance judged by those who will use the findings to take action and make decisions |
| Ultimate test of value is contribution to knowledge. | Ultimate test of value is usefulness to improve effectiveness. |
Source: Patton, Michael Quinn (2014). Evaluation Flash Cards: Embedding Evaluative Thinking in Organizational Culture. St. Paul, MN: Otto Bremer Foundation, ottobremer.org. https://ottobremer.org/news_stories/evaluation-flash-cards/
2. Evaluation and research as mutually independent
A quite different way of thinking about research and evaluation sees them as two unrelated variables that are not mutually exclusive . An activity can be BOTH research and evaluation – or neither. Research is about being empirical. Evaluation is about drawing evaluative conclusions about quality, merit or worth.
Research that is not evaluation involves factual description without judgements about quality – for example, census data, interview data which collects descriptions.
Evaluation that is not research involves making evaluative judgements without systematic collection of data – for example a connoisseur evaluator who produces a judgement without carefully gathering data.
Where they overlap is where the evaluative conclusions have been based on systematic data collection and analysis.
3. Evaluation as a subset of research
Other types of research (which are not evaluation) include basic research, applied research (which does not include evaluative conclusions.
This sees evaluation as a sub-set of research. For example, Robert Endias in a discussion on EVALTALK said:
Funny how this issue keeps coming up from time to time. Seems to me to be quite simple. Here's my view.
Research is a descriptive process engaged in for learning purposes, asking such questions as "What is/was?" or "What are/were the differences between?" or "What happens/happened when certain conditions are/were"?
Evaluation is a judgmental process, involving the assessment of findings/observations against standards, for the purpose of making decisions, asking such questions as "What is/was good?" or "Which is/was the better?" or "What conditions are the best to nurture to produce desired results?"
Doing research does not necessarily require doing evaluation. However, doing evaluation always requires doing research.
Of course, we can go on and on about how values and judgments infuse all human acts, including the act of research, and certainly the act of evaluation. Nevertheless, the basic purpose of research is to observe and learn, while the basic purpose of evaluation is to assess and decide.
Robert Endias (1998) Research vs. Evaluation on American Evaluation Association Discussion list EVALTALK. Click here for information on how to access EVALTALK.
Evaluation involves several different activities, including framing the purpose and scope of the evaluation, formalising decision making processes, deciding who will do what roles, gathering and analysising data, reporting and supporting use.
4. Research as a subset of evaluation
This view sees that research (gathering empirical evidence) is one of the tasks involved in doing an evaluation. Other tasks include: clarifying or negotiating the primary intended users and their primary intended uses identifying key evaluation questions; clarifying or negotiating resources to answer the questions; supporting use of findings.
Which framing should we use?
Each of these framing can be useful for particular purposes.
The dichotomy between research and evaluation can be useful where it is important to highlight particular features of the evaluation process. For example, Michael Patton, in an earlier discussion on EVALTALK, referred to situations where the primary intended users of an evaluation were worried about the evaluation because of their conceptualisation and previous experience with research:
Their perceptions of research tend to be that it is academic , laborious, esoteric, driven by publishing needs, and irrelevant. Now, in the face of such perceptions, where they exist, one could respond with examples of research that do not fit this stereotype. I take a different approach. I prefer to position evaluation as different from research. The basis of the distinction I make derives from contrasting standards.
...
The Joint Committee Standards of Evaluation provide quite different criteria for judging evaluations that are not typically applied to research qua research, namely, utility and feasibility (practicality).
...
I find in my practice that my clients take comfort from and appreciate having evaluation distinguished from research. They understand that research methods will be used, but the purpose of evaluation is different from research, the timelines are often quite different, and the intended uses are dramatically different, as are the primary intended audiences for findings.
In other situations, other framings are more helpful. In a conversation this week at my university, I was advised to “not use the word evaluation” to refer to the work I do. Many of my colleagues simply cannot understand how I can be a serious researcher if I am focusing on evaluation – which they associate with either low-quality contracted research, designed to support a previously made decision or action, or simply developing measures for achievement of stated objectives. In this situation it is more helpful for me to frame the discussion in terms of mutually independent terms, and to emphasise that the work I do conducting evaluations is in the overlap area which is both evaluation AND research.
In an organisation looking at the portfolio of knowledge management activities it is funding, it can be useful to think of evaluation as a subset of research, so they can discuss how to invest in a range of types of research, including some which is not explicitly evaluative and some which is.
When planning an evaluation, it can be useful to think of research as a subset of evaluation so that attention is paid to the processes of framing and managing an evaluation as well the specific research methods used to gather and analyse data. The BetterEvaluation Rainbow framework explicitly tries to bring issues about these wider tasks in evaluation into the planning for evaluations.
So how do you use these terms? What are the advantages of framing them in this way? Are there other terms that would be useful to explore?
'Week 19: Ways of framing the difference between research and evaluation' is referenced in:
Blog