Week 19: Ways of framing the difference between research and evaluation

Patricia Rogers's picture 9th May 2014 by Patricia Rogers
Also Available In: Portugues

One of the challenges of working in evaluation is that important terms (like ‘evaluation’, ‘impact’, ‘indicators’,  ‘monitoring’  and so on ) are defined and used in very different ways by different people.   Sometimes the same word is used but to mean quite different things; other times different words are used to mean the same thing. And, most importantly, many people are simply unaware that other people use these words in these different ways.

Since BetterEvaluation seeks to support discussion and learning across organisational, sectoral, and disciplinary boundaries, it is important for us to find ways to understand each other. 

This is the first in what we plan to be a series about key terms in evaluation.  It sets out four different ways of thinking about research as compared to evaluation – and when these might be useful ways of using them. 

A perennial question on discussion forums, there are surprisingly different ways of thinking about how evaluation is different to research.  These can have important implications for how we plan, manage, conduct and train people to do evaluation – and communicate to others about the process and practice of evaluation.

1. Evaluation and research as a dichotomy

One of the most common ways of thinking about the difference between research and evaluation is as a dichotomy - two distinct and mutually exclusive categories.  Research is seen as more interested in producing generalisable knowledge, more theoretical, more controlled by the researchers – evaluation is seen as more interested in specific, applied knowledge, and more controlled by those funding or commissioning the evaluation.

For example, the new Evaluation Flash Cards developed by Michael Quinn Patton for the Otto Bremer Foundation includes the following flashcard on “Evaluation versus Research”.

Research Evaluation
Purpose is testing theory and producing generalizable findings. Purpose is to determine the effectiveness of a specific program or model.
Questions originate with scholars in a discipline. Questions originate with key stakeholders and primary intended users of evaluation findings.
Quality and importance judged by peer review in a discipline. Quality and importance judged by those who will use the findings to take action and make decisions
Ultimate test of value is contribution to knowledge. Ultimate test of value is usefulness to improve effectiveness.

Patton, Michael Quinn (2014). Evaluation Flash Cards: Embedding Evaluative Thinking in Organizational Culture. St. Paul, MN:  Otto Bremer Foundation, ottobremer.org. http://www.ottobremer.org/sites/default/files/fact-sheets/OBF_flashcards_201402.pdf

2. Evaluation and research as mutually independent

A quite different way of thinking about research and evaluation sees them as  two unrelated variables that are not mutually exclusive .  An activity can be BOTH research and evaluation – or neither.  Research is about being empirical.  Evaluation is about drawing evaluative conclusions about quality, merit or worth.

Research that is not evaluation involves factual description without judgements about quality – for example, census data, interview data which collects descriptions.

Evaluation that is not research involves making evaluative judgements without systematic collection of data – for example a connoisseur evaluator who produces a judgement without carefully gathering data.

Where they overlap is where the evaluative conclusions have been based on systematic data collection and analysis.

 3. Evaluation as a subset of research

Other types of research (which are not evaluation) include basic research, applied research (which does not include evaluative conclusions.

This sees evaluation as a sub-set of research.  For example, Robert Endias in a discussion on EVALTALK said:

Funny how this issue keeps coming up from time to time.  Seems to me to be quite simple.  Here's my view.

Research is a descriptive process engaged in for learning purposes, asking such questions as "What is/was?" or "What are/were the differences between?" or "What happens/happened when certain conditions are/were"? 

Evaluation is a judgmental process, involving the assessment of findings/observations against standards, for the purpose of making decisions, asking such questions as "What is/was good?" or "Which is/was the better?" or "What conditions are the best to nurture to produce desired results?" 

Doing research does not necessarily require doing evaluation.  However, doing evaluation always requires doing research.

Of course, we can go on and on about how values and judgments infuse all human acts, including the act of research, and certainly the act of evaluation. Nevertheless, the basic purpose of research is to observe and learn, while the basic purpose of evaluation is to assess and decide.

Robert Endias (1998) Research vs. Evaluation on American Evaluation Association Discussion list EVALTALK​. Click here for information on how to access EVALTALK.

Evaluation involves several different activities, including framing the purpose and scope of the evaluation, formalising decision making processes, deciding who will do what roles, gathering and analysising data, reporting and supporting use.

4. Research as a subset of evaluation

This view sees that research (gathering empirical evidence) is one of the tasks involved in doing an evaluation.  Other tasks include: clarifying or negotiating the primary intended users and their primary intended uses identifying key evaluation questions; clarifying or negotiating resources to answer the questions; supporting use of findings.

Which framing should we use?

Each of these framing can be useful for particular purposes.

The dichotomy between research and evaluation can be useful where it is important to highlight particular features of the evaluation process.  For example, Michael Patton, in an earlier discussion on EVALTALK, referred to situations where the primary intended users of an evaluation were worried about the evaluation because of their conceptualisation and previous experience with research:

Their perceptions of research tend to be that it is academic , laborious, esoteric, driven by publishing needs, and irrelevant. Now, in the face of such perceptions, where they exist, one could respond with examples of research that do not fit this stereotype. I take a different approach. I prefer to position evaluation as different from research. The basis of the distinction I make derives from contrasting standards.
...
The Joint Committee Standards of Evaluation provide quite different criteria for judging evaluations that are not typically applied to research qua research, namely, utility and feasibility (practicality).
...
I find in my practice that my clients take comfort from and appreciate having evaluation distinguished from research. They understand that research methods will be used, but the purpose of evaluation is different from research, the timelines are often quite different, and the intended uses are dramatically different, as are the primary intended audiences for findings.

In other situations, other framings are more helpful. In a conversation this week at my university, I was advised to “not use the word evaluation” to refer to the work I do. Many of my colleagues simply cannot understand how I can be a serious researcher if I am focusing on evaluation – which they associate with either low-quality contracted research, designed to support a previously made decision or action, or simply developing measures for achievement of stated objectives. In this situation it is more helpful for me to frame the discussion in terms of mutually independent terms, and to emphasise that the work I do conducting evaluations is in the overlap area which is both evaluation AND research.

In an organisation looking at the portfolio of knowledge management activities it is funding, it can be useful to think of evaluation as a subset of research, so they can discuss how to invest in a range of types of research, including some which is not explicitly evaluative and some which is.

When planning an evaluation, it can be useful to think of research as a subset of evaluation so that attention is paid to the processes of framing and managing an evaluation as well the specific research methods used to gather and analyse data.  The BetterEvaluation Rainbow framework explicitly tries to bring issues about these wider tasks in evaluation into the planning for evaluations.

So how do you use these terms? What are the advantages of framing them in this way? Are there other terms that would be useful to explore? Let us know in the poll or comments below.

Poll

52 Weeks of BetterEvaluation. Click here to view past features

A special thanks to this page's contributors
Author
CEO, BetterEvaluation.
Melbourne.

Comments

Patricia Rogers's picture
Patricia Rogers

Thanks for these two additional examples of how the terms 'research' and 'evaluation' can have very different meanings.  There are many different types of evaluation needed for effective organisations and programs, including product evaluation, personnel evaluation, as well as program and policy evaluation.  

And, since each of these has particular requirements, we need to be thoughtful about taking a method from one situation and applying it to another.

Mulenga's picture
Cynthia Mulenga

An interesting topic indeed. Thanks for this blog, Patricia.  Only yesterday I was engaged in a debate with a colleague over the difference between evaluation and research is.

If I must choose amongst the four categories, I would lean towards evaluation being a subset of research... maybe even call it evaluative research.

I like to think that evaluation and research are two separate and different "things". I look at research as a way of obtaining data or information while evaluation is what I do with that data or information - I make judgements of worth and value.

Patricia Rogers's picture
Patricia Rogers

Ah yes - monitoring and evaluation.  Another case where the terms can be used in different ways for particular purposes.  This would also be good to explore.

BernadetteWright's picture
Bernadette Wright

I tend to use the words evaluation, evaluation research, and research inter-changeably, because in my experience many evaluators and evaluation users use them inter-changeably. For example, many studies that evaluate interventions are not clearly labelled as "evaluations". Many RFPs for evaluations ask for "research".

If I need to explain how evaluation research is different from other research that people may be familiar with, I would say that it's different from purely theoretical work because the purpose of evaluation is to provide the best possible information to improve whatever is being evaluated.

Patricia Rogers's picture
Patricia Rogers

That's what many people find, Elaine that the different definitions are useful for different purposes and situations.  Please do share it with your blog readers and let us know what they think.  And good luck with you research - it's terrific to have PhDs being conducted on evaluation.

 

Anonymous's picture
Nader Abouzeina

I have worked in the field of scientific research for a few years. Sometimes we worked on completely new tasks, while other times we worked on testing the validity and reliability of other people's work (what I consider evaluation). Yet, I was never trained in evaluation, we used the same methods of research when we conducted evaluations, which I now think is wrong. Do you think research teams should be trained in evaluation, and should research teams include trained evaluators to monitor the quality of the evaluation conducted?

Anonymous's picture
Jennifer Wallin

As a graduate student taking her first class on such a topic, I found your blog to be very clear in comparing/ contrasting the definitions of research and evaluation. The clearest module to me, and the one that seems to reflect my current course of study, appears to be module #2, where research and evaluation are indeed separate entities but will overlap in purpose and practice when practical application of the research would be ready for employ.

I will be sure to bookmark this page for further reference. Thanks for the post.

Jennifer Wallin

Anonymous's picture
Francisco Reynoso

Yes, a great way to think about evaluation and how it differs with research is that evaluation is political. Its findings are commonly used for decision making. Research, on the other hand, is used to build on prior knowledge.

Anonymous's picture
Lynn Hagwood

I am a current student at Arizona State University in the Master's program for Educational Technology.  I truly appreciate your blog post and find your information that you shared quite insightful when differentiating between research and evaluation.  I feel that many people group the two together or use the words interchangably and you provide a clear explanation of the differences that helps the reader discern between the two terms and clarifies that the two words are not synonomous.  Thank you for your enlightenment!

Anonymous's picture
Kamina Hall

I had not considered including evaluation as a subcategory of research or the other way around. This could indeed be a useful mindset in situations where an evaluative position comes under attack.

I am just wondering out loud though, do researchers who adhere to the transformative paradigm not change the specific program they are researching in the process of researching it? If so, then does this push this type of research even closer to the evaluation side? 

Anonymous's picture
Elizabeth Minatee

Thank you for this post Patricia,

I'm a graduate student at Mary Lou Fulton School of Education at Arizona State and we were debating the differences and similarities between research and evaluation. 

Our textbooks align with the part where you said, evaluation is about drawing evaluative conclusions about quality, merit or worth. 

Mertens believes that research “provides information about the need for, improvement of, or effects of programs or policies” (Mertens, 2020, p.50) without the political contexts and political maneuvering that is involved with evaluations.

I see evaluation as a type of research because you can't evaluate something without first conducting research and gathering data. However, evaluating a program does require judgment on the program's quality, merit, or worth. 

This post was very useful in summarizing the different views about contrasting research and evaluation.

 

Anonymous's picture
fareeha Waqas

Hi Ms. Rogers

I enjoyed reading your blog and especially the Venn diagrams to clarify the perspectives about evaluation and research. I agree that the interconnection of evaluation and research can wear multiple hats depending on the purpose. It is important to keep in mind that research generates generalizable knowledge and evaluation is driven by a need to measure effectiveness. Despite all the differences, they both may run in parallel formally or informally.

 

Anonymous's picture
Kou Yang

As someone who doesn't work in the science field I've always had a general sense of the terms "research" and "evaluation" Whether it was for a school project or work-related event, those terms are thrown around so much that their meanings are often intertwined and can sometimes feel similar or at least related to each other. I had never really given it much thought until now. The quote by Robert Endias about the end-product of each term really sums it up nicely and cleared up any confusion I had about the differences between the two.

Anonymous's picture
Jeffrey Jackson

This is a great lesson on the differences in research and evaluation. Research being associated with "testing" and evaluation associated with "determining" help me to delineate between the two. I had previously thought of research as a subset of evaluation and this article makes it even more clear for me. 

Anonymous's picture
Jeffrey Jackson

This article really helps me to differentiate between research and evaluation. Relating research to "testing" and evaluation to "determining" sums it up well for me.

I had previously related research as a subset of evaluation and that belief is reinforced with this article. Then the overlap with evaluation based on systematic research is clear.

Anonymous's picture
Allison Snetselaar

Thank you for this post, Patricia. I am currently enrolled in a graduate level course on educational research and evaluation, and your article was an interesting read that helped to clarify my understanding. I agree with your take on the usefulness of research vs. evaluation to distinguish the two terms. As you stated, “Research is about being empirical. Evaluation is about drawing evaluative conclusions about quality, merit, or worth”. 

What I hadn’t previously thought about that really improves my understanding is WHO is conducting research vs. evaluation. I now see that research is often done by scholars while evaluations are conducted by stakeholders in the field. Something that is of interest to me is finding a study that is both research and an evaluation, following the mutually independent model, which is conducted by an expert who is also a stakeholder. Finding such a study where so much overlap in definitions would provide an interesting perspective with meaningful take-aways.

Anonymous's picture
Lauren Erlick

I am a graduate student at Arizona State University, and have been investigating the differences and similarities between research and evaluation.  After reading several articles about the two terms I can understand what you meant by the immense amount of terminology used to describe the terms.  That is why I really enjoyed reading your article as it provided clear cut examples of the differences between the two words as well as noting that the terms sometimes can overlap.  The visual diagrams helped me to gain a grasp of how many different ways the two terms are related and why the research and evaluation processes can get confusing for someone to initially understand.  In my work experience we have used the words interchangeably, however after taking courses through the University and reading this article I can see how they are different.

Anonymous's picture
Mitchell LaCombe

I really appreciate all the detail in this post that helps define the differences between evaluation and research. The flashcards were extremely helpful and I feel that I need to make those part of my studying in order to nail this topic. 

Additionally, explaining how evaluation and research can be a subset of either was quite interesting, the visuals were really helpful as well. 

Thank you so much for this information as it has helped me tremendously. 

Mitch

Anonymous's picture
Jessica Neiweem

Hi Ms. Rogers,

I am a Masters student in Learning Design & Technology at Arizona State University, and as part of my Introduction to Research & Evaluation in Education course I was directed to read your article. I appreciate the succinctness with which you address the age-old question of "What is the difference between research and evaluation?" as this is an important distinction for those in academic contexts to understand. In particular, I like what you said about there being a need to break down the assumptions that exist about both research and evaluation, as each term seems to carry particularly negative connotations depending on who you ask. As with anything else, a little understanding and common ground can help reduce the likelihood that people will misjudge something, and after reading your article I better understand that research and evaluation are more similar than they are different. Both are about advancing knowledge. Both are about helping others find the truth about something. And while evaluation has the added goal of recommending a particular action to a particular group of stakeholders, robust and thorough research can assist an evaluator in conducting a fair and objective evaluation. In short, thanks for showing your audience that research and evaluation are friends, not enemies!

Sincerely,

Jessica Neiweem

Anonymous's picture
Jessica Neiweem

Hi Ms. Rogers,

I am a Masters student in Learning Design & Technology at Arizona State University, and as part of my Introduction to Research & Evaluation in Education course I was directed to read your article. I appreciate the succinctness with which you address the age-old question of "What is the difference between research and evaluation?" as this is an important distinction for those in academic contexts to understand. In particular, I like what you said about there being a need to break down the assumptions that exist about both research and evaluation, as each term seems to carry particularly negative connotations depending on who you ask. As with anything else, a little understanding and common ground can help reduce the likelihood that people will misjudge something, and after reading your article I better understand that research and evaluation are more similar than they are different. Both are about advancing knowledge. Both are about helping others find the truth about something. And while evaluation has the added goal of recommending a particular action to a particular group of stakeholders, robust and thorough research can assist an evaluator in conducting a fair and objective evaluation. In short, thanks for showing your audience that research and evaluation are friends, not enemies!

Sincerely,

Jessica Neiweem

Anonymous's picture
Lauren S

Hi Patricia,

I found your blog to be well framed and thoroughly written to address the areas of research vs evaluation. I particularly like your responses in the comment section that communicate the challenge of various definitions used in the workplace when discussing research and evaluation, particularly as it arises with programming in RFPs. To be honest, I actually didn't consider that people used the terms interchangeably, as I have always felt they were two very distinct activities, however, after reading the comments about language and definitions, I can see how the confusion can set in for those who have minimal exposure. For example, I do recall a time when I used the words assessment and evaluation interchangeably. Thank you for this post!

Anonymous's picture
Karina Powell

Hello,

I am a student at Arizona State University, getting my master's degree in Educational Technology and Instructional Design, and we are completing a course on research and evaluation. 

Thank you so much for you article.  I've been reading text after text about research and evaluation and your article gave me a greater understanding of the difference! When your article mentioned that research does not necessarily require doing evaluation but evaluation always requires doing research, helped me keep them into the two categories they belong. 

Thank you. 

Anonymous's picture
Mirvat Farah

Hello, I wanted to comment on your interpretations of the differences between research and evaluation. Right away I noticed your visuals as very impactful and are a quick reference to the content you are addressing in each section. I felt that it provided even more support to your findings. For visual learners (myself included) this is a 'simple' representation for a complex topic! Your clarifications reminded me that research requires peer reviews to ensure quality. While the quality of an evaluation is judged by the stakeholders who will use the findings to make decisions.  To answer your question, I still think it is a difficult to frame research vs evaluation only one way. I agree when you that it can be framed based on the situation. Thank you for the interesting read Patricia! 

Anonymous's picture
Jeffrey Garcia

Hi. I personally run into issues using terminology attributed to either research or evaluation interchangeably. You make a good point of looking at these as two separate entities, a dichotomy. I was unaware of the cultural research hierarchy such that evaluation lends itself to be a lesser form of academia when evaluation can serve as meaningful a purpose or more significant a purpose than some research in terms of overall viability and application for the greater good.

Anonymous's picture
Jeffrey Garcia

Hi. I personally run into issues using terminology attributed to either research or evaluation interchangeably. You make a good point of looking at these as two separate entities, a dichotomy. I was unaware of the cultural research hierarchy such that evaluation lends itself to be a lesser form of academia when evaluation can serve as meaningful a purpose or more significant a purpose than some research in terms of overall viability and application for the greater good.

Anonymous's picture
Van Paraskevas

I greatly enjoyed reading the article and loved how the relationship between research and evaluation was explained. I've always thought of the relationship between the two as Robert Endias stated "Doing research does not necessarily require doing evaluation.  However, doing evaluation always requires doing research" (1998).

Conversely, I hadn't done much thought about the reverse where research can be a subset of evaluation though it makes a great deal of sense as you've expressed it here. Loved the article!

Anonymous's picture
Lisa C

Hello,

I am a graduate student in the Learning Design and Instructional Technologies program at Arizona State University. This week we are learning about the differences of research and evaluation. I enjoyed reading your Week 19 Blog and found the visuals that you included to be very helpful. Framing research and evaluation into the four main areas of: evaluation and research as a dichotomy, evaluation and research as mutually independent, evaluation as a sub-set of research, and research as a sub-set of evaluation was very useful to me when forming my opinion on the differences between research and evaluation. I also agree with your statement that, each of these framing can be useful for particular purposes."

Patricia Rogers's picture
Patricia Rogers

Thank you for all these thoughtful comments on research and evaluation.  I'm glad you found this useful.  What I particularly appreciate is how you've related these different ways of framing research and evaluation to your particular situations, and thought about the potential advantages and disadvantages of each option.  It makes it clear that there is no one best way to frame research and evaluation - and that individuals and organisations have very different ways of looking at them.  For example, while it is true that the quality of some evaluations is usually judged by their intended users, some organisations include a formal process of peer review which is more like the traditional research approach. And while it is true that most evaluations are intended to have a practical purpose, the same can be said of applied research.      

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.