How do we ensure we address all the important aspects of an evaluation when we’re planning it? How do we manage to consider the different options without being overwhelmed?
This week I was pleased to join over 500 people in the first in our Coffee Break Webinar series with the American Evaluation Association to explore these issues.
In this blog post I discuss five ways I've used the rainbow framework for planning, managing and conducting evaluation, and answer some of the questions which were asked in the webinar.
You'll also find links to the recording of the webinar, which includes the slides, links to two free downloadable tools
View the webinar below (20 minutes) or in YouTube:
For a full overview of the public BetterEvaluation series of AEA Coffee Break webinars, go to http://betterevaluation.org/events/coffee_break_webinars_2013 where you can register for free to attend upcoming webinars and watch recordings of completed ones.
I’ve found that the Rainbow Framework can help both evaluators and commissioners of evaluations to make thoughtful choices that are appropriate and feasible for their situation – and makes it easier for them to make sense of the many M & E resources that are available.
The framework outlines the key tasks involved in any evaluation and organizes them into seven clusters:
- Define: Develop a description of what is to be evaluated and how it is understood to work
- Frame: Set the parameters of the evaluation – its purposes, primary intended users, key evaluation questions and the criteria and standards to be used
- Describe: Collect and analyze data to answer descriptive questions about what has happened
- Understand Causes: Collect and analyze data to answer causal questions about what has produced outcomes and impacts
- Synthesize: Combine data to form an overall assessment of the merit or worth of the intervention, or to summarize evidence across several evaluations
- Report and Use: Develop and present findings in ways that are useful for the intended users of the evaluation, and support them to make use of them
- Manage: Manage an evaluation (or a series of evaluations), including deciding who will conduct the evaluation and who will make decisions about it
I’ve already been using the Rainbow Framework in five important ways:
1. Design and plan an evaluation
Working through the clusters and related tasks can give you the confidence and clarity you need to plan a good evaluation – whether you’re new to evaluation or have considerable experience. Systematically considering all the why, what, when, who and how questions also helps with transparency - developing agreements among key stakeholders about which options are best.
2. Check the quality of an ongoing evaluation
At various moments in the evaluation, you can return to the framework and check that you are on track to deliver a transparent, appropriate and good quality evaluation. If you have encountered a tricky situation of confusion or disagreement, then the framework can help you identify what the issue is that needs clarifying and can offer some options.
3. Commission and manage an evaluation
When developing the Terms of Reference for an evaluation, the framework can help to make them clear and focused. When reviewing proposals, it can be used to assess their clarity and quality. Also, when managing an ongoing evaluation, the framework can help commissioners to keep a tab on core questions (see number 2 above).
4. Embed participation thoughtfully in evaluation
The aim of ‘engaging all stakeholders’ in evaluation is often not feasible or necessary, yet participatory evaluation should be more than just asking people for their opinions. The framework can be used to consider who needs to be involved in which stage and how, plus how to make this possible, to plan for an appropriately and meaningfully participatory evaluation.
5. Develop evaluation capacity
The framework can be used by organizations to map existing skills and knowledge, in order to identify gaps and options for filling those gaps. It can also help an organization gain insight into which aspects of planning an evaluation need to be strengthened. For individual evaluators, the framework can be useful as part of a lifelong learning endeavor. None of us know all the 200+ evaluation options described on the BetterEvaluation website. There is always more to learn. By scanning the clusters and tasks, evaluators can identify areas where they would like to bolster their own capacity and find clear options for doing this.
Questions from the webinar
Does the framework exist in other languages?
We are currently working on this. We’re delighted to announce that EvalMENA is working with us to develop Arabic subtitles for all eight webinars in the series, which will provide an introduction to the whole framework. Stay tuned for more news about languages – or contact us if you’d like to help with a particular language.
What processes are in place for quality control?
The material on BetterEvaluation is created by identified authors with experience in evaluation and expertise in using particular options. Where possible pages are written by experts who have used the option extensively or by drawing on advice from experts. All pages show the author, reviewer and steward (if active).
For example,we are delighted that our authors include Angela Ambroz and Marc Shotland, JPAL(Abdul Latif Jameel Poverty Action Lab) who are currently revising the page on RCTs (Randomized Controlled Trials), drawing on their long experience with this approach. And that our reviewers include Werner Ulrich, Martin Reynolds and Bob Williams who reviewed and improved our page on Critical Systems Heuristics, drawing on their combined experiences in developing and exploring the approach.
What is unique about the Rainbow Framework?
The ideas underpinning the framework will be very familiar to people who have been trained in evaluation. What the framework does is to embed what we know about good practice in evaluation into a simple schema that makes it accessible to evaluation managers and community members who have not been trained in evaluation.
The rainbow metaphor represents the principle of diversity and inclusion that underpins BetterEvaluation – we promote thoughtful selection of the appropriate options for a particular situation, and therefore include all evaluation options and a space for constructive and respectful discussion about which ones should be used when and where.
This blog post is part of a series of eight posts covering the BetterEvaluation Framework and presenting the recordings of eight corresponding webinars hosted by the American Evaluation Association. The full series of posts is below.
|1. Using the Rainbow Framework, Irene Guijt||5. Understanding causes, Jane Davidson|
|2. Defining what needs to be evaluated, Simon Hearn||6. Weighing the data for an overall evaluative judgment, Patricia Rogers|
|3. Framing the evaluation, Patricia Rogers||7. How can evaluation make a difference?, Simon Hearn|
|4. Choosing methods to describe activities, results and context, Irene Guijt||8. Manage an evaluation or evaluation system, Kerry Bruce|