Using logic models and theories of change better in evaluation
Many evaluations include a process of developing logic models and theories of change – an explanation of how the activities of a program, project, policy, network or event are expected to contribute to particular results in the short-term and longer-term.
They have been used for many years - versions can be seen in Carol Weiss’ 1972 book "Evaluation research: methods for assessing program effectiveness" - and they have been mainstreamed in many organisations as an essential component of planning and evaluation under various labels, including program theory, programme theory, intervention logic, investment logic, outcomes hierarchy.
However, their full potential is often not met, as many people seem to think that the basic version they know is all there is, even when it doesn't really meet their needs.
We’ve explored some of these issues in previous blogs and events and so thought it would be useful to group some of these resources together under some of the common challenge areas people have with theory of change:
Developing a theory of change
The process of developing a theory of change doesn’t have to only involve a group of people writing on sticky notes. It’s often important to bring in information from research, previous evaluations, and the perspectives of those with lived experience of the program or the situation it is intended to address. And it’s important to actually have a theory – an explanation of HOW you expect activities to contribute to the intended results.
One of the common problems in using theory of change is not actually having a theory. This blog addresses this issue.
It's important to make sure that the theory of change actually has a theory about how change will come about - not just some boxes with arrows between activities and outcomes/impacts. So how do you choose which change theory to use?
Projects and programs that are based on an inadequate theory of change are less likely to be effective as plans and activities will not cover everything that needs to be done, and projects will be implemented when there is little chance of success.
Methods on BetterEvaluation: Processes for developing a programme theory
Articulating mental models: talking individually or in groups with key informants (including programme planners, service implementors and clients) about how they understand an intervention works.
Backcasting: working backward from a desirable future, to the present in order to determine the feasibility of the idea or project.
Five Whys: asking questions in order to examine the cause-and-effect relationships that create underlying problems.
Group model building: building a logic model in a group, often using sticky notes.
Previous research and evaluation: using the findings from evaluation and research studies that were previously conducted on the same or closely related areas.
SWOT Analysis: reflecting on and assessing the Strengths, Weaknesses,Opportunities and Threats of a particular strategy in order to discover how it can best be implemented.
Representing a theory of change
A theory of change doesn’t have to only be in the form of a pipeline of:
inputs -> activities -> outputs -> outcomes -> impacts
It can often be more useful to represent a theory of change in the form of a sequence of results where activities can occur along the chain ( an outcome hierarchy) or a triple column/row version which shows activities and other factors visually. And there are lots of useful technologies that can be used.
This blog discusses the work of Carol Weiss, who showed how useful it can be to focus on teasing out the different possible causal paths between program activities and their outcomes and gives some examples and tips for drawing logic models.
In 2013, Simon Hearn presented the second of eight AEA Coffee Break webinars, introducing the DEFINE component of the BetterEvaluation Rainbow Framework. This blog responds to the many questions that were asked by the participants - in particular there was a great question about non-linear logic models.
Methods on BetterEvaluation: Ways of representing programme theory in a logic model
- Tiny Tools Results Chain: mapping both positive and negative possible impacts from an intervention
- Logframe: designing, executing and assessing projects by considering the relationships between available resources, planned activities, and desired changes or results.
- Outcomes hierarchy (also known as a theory of change or an outcomes chain): showing a series of outcomes leading up to the final impacts of a project.
- Realist matrix: focusing on one of the steps in an outcomes chain and then identifying the mechanism involved in producing the outcome and the contexts within which this mechanism operates.
- Results chain (also known as a ‘pipeline model’): showing a programme as a series of boxes inputs-> activities-> outputs -> outcomes -> impacts
- Triple column: showing an outcomes hierarchy in the central column
Using a theory of change
A theory of change is often used for planning a program or project, developing a clearer and more plausible plan but sometimes its benefits for monitoring and evaluation are not realised.
Here are some ways to use it:
- Guide data collection by focusing on what is needed in terms of measures, indicators or metrics of intended outcomes.
- Identify which outcomes are likely to be evident during the life of the evaluation
- Identify other sources of evidence that can support later causal links - for example, early childhood programs are often evaluated well before the effects on children's education and employment can be seen, but these evaluations can draw on evidence from research and evaluations about the likely positive impacts of improving literacy, secure attachment and emotional intelligence.
- Explain whether failure to achieve intended results is due to implementation failure or to theory failure - by connecting information about processes with information about results across cases or sites
- Strengthen causal inference by identifying evidence that is either consistent with or challenges the theory of change
- Support generalisation by identifying what works for whom in what context
- Support synthesis across different studies with a common theory of change
Many development programme staff have had the experience of commissioning an impact evaluation towards the end of a project or programme only to find that the monitoring system did not provide adequate data about implementation, context, baselines or interim results. This Methods Lab guidance note by Greet Peersman, Patricia Rogers, Irene Guijt, Simon Hearn, Tiina Pasanen and Anne L. Buffardi has been developed in response to this common problem.
This blog by E. Jane Davidson and Patricia Rogers discusses how program theory can support analysis of causal attribution and contribution even when there is not a credible counterfactual. (See the BetterEvaluation page on Understanding Causes for more information on these methods)
Need more support?
For those of you looking for more support or advice for using theories of change, we're offering BetterEvaluation members the chance to submit a question or challenge that they have in relation to creating or using theory of change for review by the BetterEvaluation team.
We'll be selecting a number of these questions to be de-identified and answered by Patricia Rogers in next newsletter's blog. Contact us privately with your question, or leave it as a comment below.
Patricia Rogers will also be exploring these issues in the next few months in two courses on advanced use of program theory - TEI Washington, during their 2017 July Program, and ANZSOG Melbourne, on August the 16th.
And of course, as always, we welcome your thoughts, suggestions, and resources on this topic, so if you have something to share, let us know.