An evaluation design describes how data will be collected and analysed to answer the Key Evaluation Questions.
There are different pathways for you as manager depending on who will develop the evaluation design. In most cases your evaluator will develop the evaluation design. In some cases you will – if you have evaluation expertise and/or the evaluation design has already been developed (for example, in an evaluation that is intended to match an earlier evaluation).
- The nature of what is being evaluated
In particular whether there are complicated or complex aspects that need to be addressed in the evaluation, and other particular challenges such as delays before impacts are evident or barriers to collecting accurate data. Read more.
- The nature of the evaluation
In particular the types of Key Evaluation Questions that are being asked, when the answers are needed. Read more.
- Available resources and constraints
Resources including money, existing data, expertise, technical equipment. Constraints including requirements to use certain common indicators, limits to availability of key informants or barriers to accessing existing data. Read more.
If an EVALUATOR will develop the evaluation design
- Engage a competent evaluation expert (internal, external or a combination) (See advice).
- Work with the expert(s) to ensure they understand important factors that should be taken into account in the evaluation design
- The design should provide details of how data will be collected analysed. It is often useful to do this in the form of an Evaluation Matrix which shows how each Key Evaluation Question will be answered.
If YOU (as manager) will develop the evaluation design
- Understand important factors that should be taken into account in the evaluation design
- Develop an evaluation design that addresses these important factors.
- Summarise the design in the form of an Evaluation Matrix which shows how each Key Evaluation Question will be answered.
Subsequently, arrange for a technical review of the evaluation design and arrange for a review of the design by the evaluation management structure (e.g., steering committee). Ideally this will include representation from primary intended users.
Arranging technical review of the evaluation design
Before finalizing the design, it can be helpful to have a technical review of it by one or more independent evaluators. It might be necessary to involve more than one reviewer in order to provide expert advice on the specific methods proposed, including specific indicators and measures to be used. Ensure that the reviewer is experienced in using a range of methods and designs, and well briefed on the context, to ensure they can provide situation specific advice.
Arranging review of the design by the evaluation management structure
In addition to being considered technically sound by experts, it is essential for the evaluation design to be seen as credible by those who are expected to use it.
Get formal organisational review and endorsement of the design by an evaluation steering committee (see section for possible structures, processes and terms of reference for a steering committee)
Where possible do data rehearsal of possible findings with primary intended. This is a powerful strategy for checking the appropriateness of the design by presenting mock-ups of tables, graphs and quotes that the design might produce. It is best to produce at least 2 different versions – one that would show the program working well and one that would show it not working. Ideally the primary intended users of the evaluation will review these and either confirm the suitability of the design or request amendments to make the potential findings more relevant and credible. (For more information see Patton, MQ (2011) Essentials of Utilization-Focused Evaluation. pp. 309-321).
You are currently here:
+ - 2. Scope the evaluation
- Clarify what will be evaluated
- Describe the theory of change
- Identify who are the primary intended users of the evaluation and what will they use it for
- Develop agreed key evaluation questions
- Decide the timing of the evaluation
- Decide whether the evaluation will be done by an external team, an internal team or a hybrid of both
- Determine the evaluator qualities
- Identify what resources are available for the evaluation and what will be needed