Manage development of the evaluation methodology

What it is: 

An evaluation design describes how data will be collected and analysed to answer the Key Evaluation Questions. 

There are different pathways for you as manager depending on who will develop the evaluation design. In most cases your evaluator will develop the evaluation design.  In some cases you will – if you have evaluation expertise and/or the evaluation design has already been developed (for example, in an evaluation that is intended to match an earlier evaluation).

Take into account the following important factors when developing an evaluation design

  1. The nature of what is being evaluated

    In particular whether there are complicated or complex aspects that need to be addressed in the evaluation, and other particular challenges such as delays before impacts are evident or barriers to collecting accurate data. Read more.
  2. The nature of the evaluation

    In particular the types of Key Evaluation Questions that are being asked, when the answers are needed. Read more.
  3. Available resources and constraints

    Resources including money, existing data, expertise, technical equipment. Constraints including requirements to use certain common indicators, limits to availability of key informants or barriers to accessing existing data. Read more.

If an EVALUATOR will develop the evaluation design

If  YOU (as manager) will develop the evaluation design

Subsequently, arrange for a technical review of the evaluation design and arrange for a review of the design by the evaluation management structure (e.g., steering committee). Ideally this will include representation from primary intended users.

Arranging technical review of the evaluation design

Before finalizing the design, it can be helpful to have a technical review of it by one or more independent evaluators.  It might be necessary to involve more than one reviewer in order to provide expert advice on the specific methods proposed, including specific indicators and measures to be used.  Ensure that the reviewer is experienced in using a range of methods and designs, and well briefed on the context, to ensure they can provide situation specific advice.

Arranging review of the design by the evaluation management structure

In addition to being considered technically sound by experts, it is essential for the evaluation design to be seen as credible by those who are expected to use it.

Get formal organisational review and endorsement of the design by an evaluation steering committee (see section for possible structures, processes and terms of reference for a steering committee)

Where possible do data rehearsal of possible findings with primary intended. This is a powerful strategy for checking the appropriateness of the design by presenting mock-ups of tables, graphs and quotes that the design might produce. It is best to produce at least 2 different versions – one that would show the program working well and one that would show it not working.  Ideally the primary intended users of the evaluation will review these and either confirm the suitability of the design or request amendments to make the potential findings more relevant and credible. (For more information see Patton, MQ (2011) Essentials of Utilization-Focused Evaluation. pp. 309-321).

You are currently here:
Click here to view full menu

Comments

There are currently no comments. Be the first to comment on this page!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.