52 weeks of BetterEvaluation: Week 20: Defining what needs to be evaluated
Whether you are commissioning an evaluation, designing one or implementing one, having - and sharing - a very clear understanding of what is being evaluated is paramount. For complicated or complex interventions this isn't always as straightforward as it sounds, which is why BetterEvaluation offers specific guidance on methods for doing this.
On Thursday last week, I presented the second of eight webinars hosted with the American Evaluation Association introducing the DEFINE component of the BetterEvaluation Rainbow Framework - the set of tasks to help define exactly what is to be evaluated. This blog presents the recording of the webinar and responds to the many questions that were asked by the participants - in particular, there was a great question about non-linear logic models and we've found a few examples to share (including one from another participant) but we'd love to hear from you if you have other examples.
You can watch the webinar, access downloadable slides, and get a full overview of the webinar series below:
Define what is to be evaluated webinar and slides
BetterEvaluation series of AEA Coffee Break webinars.
Questions from the webinar
There were a few questions asked by participants around logic models and a few more general questions:
Questions about logic models...
1) Can you talk about ways to diagram program theory (e.g. logic model, etc.) that don't just show causal relations as linear and unidirectional relationships?
I mentioned in the webinar that the Outcome Mapping approach helps build non-linear theories of change but it is not so much visual. These two examples demonstrate how people have used Outcome Mapping to enhance their logframes: OM ideas 7: The use of Outcome Mapping in value-chain development & OM ideas 6: Making ‘Evidence’ the Plural of ‘Anecdote'.
This is something we'd really like to hear from you about. One webinar participant shared their example and we've managed to find a few others, but I'm sure there are more. Leave a comment below and we'll add them to the site. Here's what we have so far:
- Evaluation of the Aboriginal Research Pilot Program: One attendee shared this evaluation report following the webinar - see page 14 for a non-linear logic model, and p.13 for the standard pipeline version (thank you to Orsolya Vaska!)
- Lifting the lens: developing a logic for a complicated policy: This article presents a standard logic model developed by the lead agency and a non-linear version developed by the evaluators to better describe simultaneous causal strands.
-
Creating program logic models: This book chapter includes a section on non-linear logic models.
-
Causal Loop Diagrams: Little-known analytical tool: Causal loop diagrams can be a good way to show complicated feedback loops.
2) If you are working on an overall project evaluation, could it be possible to define the theory of change/logic model prior to defining the evaluation questions?
Yes, working through a logic model can help identify evaluation questions. But it also works the other way around - evaluation questions can help clarify the level of detail needed in a logic model, and which aspects need to be described in more detail.
3) Are there times when certain logic model methods are more useful than others? For example, is there one that's better for evaluating an organizational culture?
Yes. Our experience has been that pipeline models (and to some extent logical frameworks) are not as useful as the outcomes hierarchy or realist matrix methods. for representing interventions with complicated aspects (lots of components) or complex aspects (emergent features) or where activities are not all at the start of the causal process but occur throughout. It would depend on the intervention and the mechanisms by which it is expected to contribute to ultimate outcomes.
4) When you have two or more outcomes, how do you create a log frame that shows different ways of getting to the same goal?
It will depend on which format you are using. For example, the DFID framework guidance (PDF) states that there should only be one outcome, covering all the changes that the intervention is seeking to bring about. To describe multiple strategies for contributing to the outcome, my suggestion would be to define multiple outputs - the effects of your activities. Beyond this, I will have to defer to logframe experts - anyone?
5) In the logframe on slide 12, in the results area (first column) you pass directly from purpose to output. Why you do not use outcomes after purpose?
For this example, we were following a basic logframe model, but there are many different variations using different terminology. For example, the DFID logframe replaced goal with impact and purpose with outcomes.
6) I think it is better to distinguish the Logframe APPROACH from the logframe matrix which you showed. All logical modes are variation on cause effect
The logical framework approach (PDF) is a facilitated process for constructing a logical framework with a group. The logical framework is the product of this but many argue (PDF) that the process is more useful than the product.
Questions about DEFINE in general...
7) What about use? Shouldn't that drive your design and be part of the issues to consider? If evaluation is not used, then why do it?
Yes, absolutely. This is covered in the FRAME cluster, which is the subject of the next webinar Framing the evaluation.
8) At what point should the stakeholders be engaged in evaluation planning?
It will depend on which stakeholders, what type of evaluation as well many other things. See the task page on understand and engage stakeholders.
9) Does each stage of the Definition process imply use of all the sample strategies? e.g. more than one program theory approach - or 4 under Unintended Consequences.
No, these are all options. Advice is provided on the BetterEvaluation website to help you choose which is more appropriate.
10) Unintended results seem to be described as negative. Can they not also be positive?
Yes, they can. You can often be pleasantly surprised! We've noted your observation that the site currently has an emphasis on options to look for negative results and will redress the imbalance by adding more on positive unintended results.
11) Could you expand a little more on the 6 Hats Thinking process?
Developed by Dr Edward de Bono, the "Six Thinking Hats" technique is a framework designed to promote holistic and lateral thinking in decision-making and evaluation. Conducted alone or in group meetings, participants – project members, key decision-makers and stakeholders – are encouraged to cycle through different modalities of thinking using the metaphor of wearing different conceptual "hats".
On the Six Hats Thinking method page, you can find an expanded description, an example of it in use, advice for using this method, and resources to guide you in using it.
12) I did not quite understand the use of the 'peak experience'. Please explain more on this.
This method provides a succinct and coherent description of a program, project or policy when it is operating at its best. This can then be used to develop a logic model (program theory) and to develop an evaluation plan to investigate how often it operates like this and how this can happen more often.
On the Peak experience description method page, you can find further information including an example, advice for choosing and using the method, and useful resources.
Resources
Defining what is to be evaluated (PDF Download): A summary of the tasks, methods, and approaches associated with defining what is to be evaluated
This is part of a series
This blog post is part of a series of eight posts covering the BetterEvaluation Rainbow Framework and presenting the recordings of eight corresponding webinars hosted by the American Evaluation Association. The full series of posts is below.
'52 weeks of BetterEvaluation: Week 20: Defining what needs to be evaluated' is referenced in:
Blog