Develop agreed key evaluation questions
Evaluation, by definition, must answer truly evaluative questions: it must ask not only ‘What were the results?’ (a descriptive question) but also ‘How good were the results?’ (an evaluative question).
Depending on the type of evaluation, causal questions also need to be addressed (to what extent were the results due to the intervention?).
An evaluation should be focused around answering a small number of high-level key evaluation questions (KEQs) which are about performance overall. Each of these key evaluation questions (KEQs) should be further unpacked by asking more detailed questions about performance on specific dimensions of merit (related to evaluative criteria such as relevance, equity, effectiveness, sustainability). The KEQs also need to reflect the intended uses of the evaluation.
Good KEQs are:
- Limited in number: 7 ± 2 questions is a good number in general. This allows for coverage of different aspects of the intervention, but is a small enough number of questions to not get overwhelmed.
- Open questions (not yes/no answers).
- Specific enough to help focus the evaluation, but broad enough to be broken down further into more detailed questions to guide data collection.
- Developed through work with primary intended users of the evaluation to develop an agreed list of key evaluation questions.
Work with primary intended users of the evaluation to develop an agreed list of key evaluation questions.
Being clear about the intended use of the evaluation and the type of evaluation needed, can help with developing appropriate Key Evaluation Questions.
The following typology can be used to classify the type of evaluation and typical questions:
- Needs analysis
- What is needed?
- What are unmet needs?
- Intervention design
- What is the best way to design the intervention?
- Monitoring
- How is it going? (regular reporting of metrics)
- Process evaluation
- Is the intervention being implemented according to plan (periodic investigations)?
- What has been done in an innovative program?
- Outcome / impact evaluation
- What results have been produced?
- What has (and has not) worked for whom in what circumstances?
- Economic evaluation
- Has the intervention been cost-effective (compared to alternatives)?
- What has been the ratio of costs to benefits?
[Source: Adapted from Owen J with Rogers P (1999). Program Evaluation: Forms and Approaches. Sydney: Allen & Unwin/London: Sage UK.]
These evaluation types are cumulative: outcome/impact evaluation needs data from process evaluation, and economic evaluation requires data from outcome impact evaluation.
The level of existing knowledge will also be important in developing appropriate evaluation questions:
- When we know what works & why, it is sensible to...
- …ask if processes are being followed (describe activities compared to an agreed standard)
- …demonstrate the value of what is being done (describe outcomes compared to an agreed statement of goals and/or needs)
- When we don’t know if it works, it is sensible to...
- …look at processes and outcomes/impacts and use an appropriate causal inference design to test the theory of change
- When we don’t know which is the best way, it is sensible to...
- …document process & context & compare performance (outcomes/impacts, efficiency)
- When we don’t know what could work, it is sensible to...
- …use action research/learning & share results (ask a series of questions about early indications of success or failure)
Product
The following item is a potential output from this sub-step. Where possible, it might be useful to research other deliverables that have also been shown to be effective.
- List of agreed key evaluation questions
Examples
- Download an example of key evaluation questions from an evaluation of the African Institute for Mathematical Sciences (AIMS)
Resources
- Good Evaluation Questions: A Checklist to Help Focus Your Evaluation
Guidance developed by the National Asthma Control Program, US Centers for Disease Control and Prevention.
- What are the key evaluation questions? (Archive link)
Guidance developed by the Office for Learning and Teaching of the Australian Government.
'Develop agreed key evaluation questions' is referenced in:
Blog
Rahmenkonzept/Anleitung
- Communication for Development (C4D) :
- Manager's guide to evaluation :
- Rainbow Framework :
Theme