Choose methods and processes
For any task in evaluation, the choice of method or process should be based on what is appropriate for your particular situation.
This means paying attention to the nature of what is being evaluated, the nature of the evaluation, the opportunities and limitations at the time, as well as who will use the evaluation, and for what purpose.
Your planning should clarify the purposes of the evaluation and the most important questions it needs to answer. Then you can choose methods and processes to answer those questions, or work with an expert who can provide advice or develop an evaluation design.
The following information is designed to help you navigate some of the key issues you'll need to work through as you plan an evaluation.
Advice for choosing methods and processes
- Choose methods or processes for every task in the evaluation (not just the design tasks)
- Analyse the types of Key Evaluation Questions (KEQ) you want to answer
- Consider your particular situation
- Review the advice provided for each method
- Aim to use a complementary mix of methods
- Create an evaluation matrix
- Check feasibility
- Get your evaluation design and plan reviewed
- Revise as necessary
1. Choose methods and processes for every task in evaluation (not just the design tasks)
It is important to think through the methods and processes you will use for doing all the tasks in an evaluation including the upfront framing work of the evaluation, the design tasks, and how you will report and support use of the evaluation findings – don’t fall into the trap of only considering the design tasks.
It is a good idea to plan the methods and processes from the beginning to ensure a good evaluation, even if you need to change the details later.
To help you find and choose appropriate evaluation methods and processes, use the Rainbow Framework.
The Rainbow Framework organises all the steps in evaluation into 7 clusters of tasks:
- Manage
- Define
- Frame
- Describe
- Understand Causes
- Synthesise
- Report and Support Use
Four of the clusters of tasks relate to methods and processes for planning and implementing the evaluation:
- Manage
- Define
- Frame
- Report and support use
Three of the clusters group methods and processes for answering different types of key evaluation questions—these are the high-level questions that an evaluation is designed to answer:
- Describe
- Understand causes
- Synthesise data
2. Analyse the types of Key Evaluation Questions you want to answer
Your evaluation plan will specify how you will collect and analyse data. It is useful to plan your data collection and analysis around a few Key Evaluation Questions (KEQs). These are high-level questions that the evaluation is intended to answer, for example, "How effective was the program?"
You will need to use different types of methods and processes for different types of evaluation questions:
Types of evaluation questions
Descriptive questions - what has happened?
For example:
- Was the policy implemented as planned?
- Did school attendance rates increase after the government abolished tuition fees at government schools?
Methods and processes to answer descriptive questions can be found in the describe section of the rainbow framework.
Causal questions - what caused these things to happen?
For example:
- Did the policy change contribute to increases in school attendance rates?
Methods and processes to answer causal questions can be found in the understand causes section of the rainbow framework
Evaluative questions - what is the overall quality of what is being evaluated?
For example:
- Taking into account the negative impacts of the policy, was the policy overall a success?
Methods and processes to answer evaluative questions can be found in the synthesise section of the rainbow framework
Action questions - what should we do?
Methods and processes to answer action questions can be found in the report and support use section of the rainbow framework
3. Consider your particular situation when choosing methods to answer the KEQs
Good evaluation is situationally appropriate.
You should take into account:
- The nature of what is being evaluated — the visibility of activities, outcomes and impacts, the time before outcomes and impacts will occur, the stage of development, and whether it has complicated or complex aspects.
-
The nature of your evaluation — the types of questions it is asking, the particular information needs and format preferences of your primary intended users
-
Constraints and available resources — including existing evidence, funding, expertise and staff time when findings are needed to inform decisions
It is likely that your evaluation will need to answer descriptive, causal and evaluative questions and will need a combination of methods to do this.
4. Review the advice provided for each method and process
For each evaluation task you will find a range of methods and processes, each with its own advantages and disadvantages.
For example, when gathering data to answer descriptive questions:
- An email questionnaire can gather data quickly — but response rates are often low, and it will not include those without technological access.
- A pen and paper questionnaire needs less technical support — but can be harder to distribute and collect and take longer to analyse.
Many methods and processes in the Rainbow Framework include advice about when it might be appropriate to choose that particular method or process, taking into account available time, expertise and other issues.
5. Aim to use a complementary mix of methods and processes
For any evaluation, you’ll need a combination of methods and processes for different aspects of the evaluation. These methods and processes are useful for completing evaluation tasks related to collecting data, analysing data, reporting data, and making decisions about the evaluation.
Wherever possible, don’t rely on a single data source but combine different data sources to reduce the risk of errors and biases and improve understanding. Using a combination of qualitative and quantitative data can improve an evaluation by ensuring that the limitations of one type of data are balanced by the strengths of another. It is important to plan in advance how these will be combined.
6. Create an evaluation matrix
When you have identified methods that might be suitable for answering Key Evaluation Questions (KEQs), create a matrix of the KEQs and selected methods.
This will help you check that the planned data collection will cover all the KEQs, see if there is sufficient triangulation between different data sources, and help you design questionnaires, interview schedules, data extraction tools for project records, and observation tools, to ensure they gather the necessary data.
Participant Questionnaire | Key Informant Interviews | Project Records | Observation of program implementation | |
---|---|---|---|---|
KEQ1 What was the quality of implementation? | ✔ | ✔ | ✔ | ✔ |
KEQ2 To what extent were the program objectives met? | ✔ | ✔ | ✔ | |
KEQ3 What other impacts did the program have? | ✔ | ✔ | ||
KEQ4 How could the program be improved? | ✔ | ✔ | ✔ |
7. Check feasibility
Limited financial resources set boundaries for what methods are feasible.
For example, a resources stocktake can help you to identify gaps between the resources you have available and what you anticipate you will need. You can then consider leveraging additional resources or adapting the evaluation to work with the available resources.
Plan the schedule of data collection, analysis and reporting and check there will be sufficient time to use these methods.
For example, key informants might not be available to meet at the time when data are needed.
Analyse the equipment and skills that will be needed to use these methods, and assess whether these are available, or can be obtained or developed.
For example, collecting questionnaire data by mobile phones will require either that each data collector has a mobile phone or that there is a reliable system for sharing these among the data collectors.
8. Get your evaluation design and evaluation plan reviewed
Build a formal review of the evaluation design (how evaluation questions will be answered) and evaluation plan (the larger framing of the evaluation including its purposes and management) into your processes. This can be done by an internal or external expert, through a mutual peer review process, or as part of the management of the evaluation.
9. Revise as needed
Given that not all can be anticipated at the start of an evaluation and that certain conditions may change during the evaluation, you may need to revisit and revise the choices you have made. In these cases, document what was changed and why, and consider and document any implications these changes may have on the evaluation product and its use.