Guidance on choosing methods and processes

There are literally hundreds of different evaluation methods and processes, and dozens of evaluation approaches that combine these into different packages, so how do you choose which ones to use?

This quick guidance page on choosing methods is designed to help you navigate some of the key issues you'll need to work through as you go through this process.

Contents

1. Choose methods and processes for every task in evaluation

Evaluation is not just about data collection. The BetterEvaluation Rainbow Framework is organised around 7 clusters of tasks needed for evaluation.

Four of the clusters relate to methods and processes for planning and implementing the evaluation:

Manage
Define
Frame
Report and support use

Three of the clusters group methods and processes for answering different types of key evaluation questions—these are the high level questions that an evaluation is designed to answer:

Describe
Understand causes
Synthesise data

You can explore different methods and processes for each task through the online version of the Rainbow Framework, or download the Rainbow Framework PDFs to read or share with others.

^Back to top

2. Analyse the types of Key Evaluation Questions you want to answer

Your Evaluation Plan will specify how you will collect and analyse data. It is useful to plan your data collection and analysis around a few Key Evaluation Questions (KEQs). These are high level questions that the evaluation is intended to answer, for example, "How effective was the program?"

You will need to use different types of methods and processes for different types of evaluation questions:

Type of question	Example	Where to find methods and processes for answering this type of question
Descriptive What has happened?	Was the policy implemented as planned? Did school attendance rates increase after the government abolished tuition fees at government schools?
Causal What caused these things to happen?	Did the policy change contribute to increases in school attendance rates?
Evaluative What is the overall quality of what is being evaluated?	Taking into account the negative impacts of the policy, was the policy overall a success?
Action What should we do?

Our task page, Specify Key Evaluation Questions, contains more information on this.

^Back to top

3. Consider your particular situation when choosing methods to answer the KEQs

Good evaluation is situationally appropriate.

You should take into account:

The nature of what is being evaluated — the visibility of activities, outcomes and impacts, the time before outcomes and impacts will occur, the stage of development, and whether it has complicated or complex aspects.

The nature of your evaluation — the types of questions it is asking, the particular information needs and format preferences of your primary intended users
Constraints and available resources — including existing evidence, funding, expertise, staff time, when findings are needed to inform decisions

It is likely that your evaluation will need to answer descriptive, causal and evaluative questions and will need a combination of methods to do this.

^Back to top

4. Review the advice provided for each method and process

For each evaluation task you will find a range of methods and processes, each with their own advantages and disadvantages.

For example, when gathering data to answer descriptive questions:

An email questionnaire can gather data quickly — but response rates are often low, and it will not include those without technological access.

A pen and paper questionnaire needs less technical support — but can be harder to distribute and collect and take longer to analyse.

On each method and processes page, you will find advice about when it might be appropriate to choose that method or process, taking into account available time, expertise and other issues.

^Back to top

5. Aim to use a complementary mix of methods and processes

For any evaluation, you’ll need a combination of methods and processes for different aspects of the evaluation. These methods and processes are useful for completing evaluation tasks related to collecting data, analysing data, reporting data, and making decisions about the evaluation.

You can get an overview of the different tasks involved in evaluation in our Rainbow Framework. You can also download PDF versions of this as a quick reference.

Wherever possible, don’t rely on a single data source but combine different data sources to reduce the risk of errors and biases and improve understanding. Check out the advice on ways of combining different types of data especially in terms of triangulating to improve validity

^Back to top

6. Create an evaluation matrix

When you have identified options that might be suitable for answering Key Evaluation Questions (KEQs), create a matrix of the KEQs and selected options.

This will help you check that the planned data collection will cover all the KEQs, see if there is sufficient triangulation between different data sources, and help you design questionnaires, interview schedules, data extraction tools for project records, and observation tools, to ensure they gather the necessary data.

	Participant Questionnaire	Key Informant Interviews	Project Records	Observation of program implementation
KEQ1 What was the quality of implementation?	✔	✔	✔	✔
KEQ2 To what extent were the program objectives met?	✔	✔	✔
KEQ3 What other impacts did the program have?	✔	✔
KEQ4 How could the program be improved?	✔	✔		✔

^Back to top

7. Check feasibility

Limited financial resources set boundaries for what options are feasible.

For example, the resources stocktake page describes a tool that helps with identifying gaps between the resources you have available and what you anticipate you will need. This prompts you to consider either leveraging additional resources or adapting the evaluation to work with the available resources.

Plan the schedule of data collection, analysis and reporting and check there will be sufficient time to use these methods.

For example, key informants might not be available to meet at the time when data are needed.

Analyse the equipment and skills that will be needed to use these methods, and assess whether these are available, or can be obtained or developed.

For example, collecting questionnaire data by mobile phones will require either that each data collector has a mobile phone or that there is a reliable system for sharing these among the data collectors.

^Back to top

8. Get your evaluation design and evaluation plan reviewed

Build into your processes formal review of the evaluation design (how evaluation questions will be answered) and evaluation plan (the larger framing of the evaluation including its purposes and management). This can be done by an internal or external expert, through a mutual peer review process, or as part of the management of the evaluation.

^Back to top

9. Revise as needed

Given that not all can be anticipated at the start of an evaluation or that certain conditions may change during the evaluation, you may need to revisit and revise the choices you have made. In these cases, document what was changed and why, and consider and document any implications these changes may have on the evaluation product and its use.

^Back to top