BetterEvaluation FAQ: How do I choose a suitable theory of change?

Patricia Rogers's picture 22nd July 2016 by Patricia Rogers

We've been getting some great questions recently - so we're starting to share our answers through these blogs.  We'd welcome any additional suggestions for how to respond, or useful resources. Just add these to the comments box below.

I’m wanting to do an impact evaluation. What kinds of theories should I use for developing a theory of change?

A theory of change can be very useful in designing an impact evaluation.  It can identify what you should collect data about and how help guide how it should be analysed and reported.  

It's important to make sure that the theory of change actually has a theory about how change will come about  - not just some boxes with arrows between activities and outcomes/impacts.

So how do you choose which change theory to use? There are a number of options and in practice it's often useful to combine these:

  1. Identify the "big t" Theories from previous research and evaluations

Previous research in the relevant area often discusses how change occurs at different levels - for individuals, families, communities and organisations. Reviewing this research can help to identify possible "big t" Theories, and give you an understanding of which might be most appropriate. 

For example, in relation to theories for behaviour change, a systematic review undertaken by Susan Michie et al. (2011) identified  83 different behaviour change theories!  

You might find one of these particularly relevant. or their summary overview which has three clusters: capabilities (skills and knowledge), motivation (positive and negative, intrinsic and extrinsic) and opportunities.

(The COM-B system - A Framework for Understanding Behaviour. Michie et al. Implementation Science, 2011, 6:42)

There might also be middle-range theories such as deterrence that have relevance. 

Sometimes there are specific change theories related to the sector.  For example if you wanted to evaluate the impact of a purchase of a defence procurement project, a review of research in the area search on Defence Policy theory might come up with a few leads.  The Common Security and Defence Policy and IR Theory gives a short overview of a few relevant theories (e.g. Liberal Institutionalism, Materialism, and realism), and a paper on Realism and the Common Security and Defence Policy also has some options.

A systematic review would identify more theories, and give you an understanding of which "big t” Theories might be most appropriate in your particular case.

  1. Look at the theories implicit in the policy documentation, including any background papers.

It’s always a good idea to go back to the project’s documentation, especially any background papers on the policy. It's particularly important to identify any theories which have been explicitly referred to in discussions or decisions.

For example, some crime prevention programs make explicit reference to  the "broken window" theory of crime prevention  - which is that signs of disorder such as unrepaired broken windows leads to increased withdrawal by local residents, and decreased informal social control, creating more opportunity for serious crime to move in. (The Center for Evidence-Based Crime Policy has more information on the theory and the evidence about whether and when it holds). 

  1. Draw out the mental models of those involved. 

Articulating mental models involves talking individually or in groups with key informants (including program planners, service implementors and clients) about how they understand an intervention works. If you can't talk with them, you might be able to find notes from discussions about how the program or project is meant to work.

Our Option Page on Articulating Mental Models has a list of questions that can be used to draw out these mental models.

  1. Logical analysis of the causal paths - how might the dots be connected?

In addition to these sources, it can be useful to logically analyse the actors and possible causal paths.  For example, in a program involving home visits by teachers, improvements to student learning might come about through changes in the teachers (for example, learning about barriers to students' learning such as lack of study spaces at home), the students (for example, developing a better relationship with the teacher), or through the parents (for example, developing a willingness to support school processes).

These different possible theories can be explored in discussions (while exploring existing mental models) and when reviewing research and project documentation.

What are the strategies you use to identify possibly relevant theories for your theory of change?

A special thanks to this page's contributors
Author
Director of BetterEvaluation/ Professor of Public Sector Evaluation, Australia and New Zealand School of Government.
Melbourne.

Comments

stevepowell99's picture
stephen powell

Hi Patricia, thanks for this! When it comes to actually drawing theory of change diagrams, just a quick plug for my (free, open-source) online app, called Theory Maker (http://theorymaker.info) - it is mentioned here on BetterEvaluation. There you can also find a growing gallery of templates and examples contributed by users.

Anonymous's picture
John Pilla

The use and selection of the theory of change also depends on the context for the evaluation. For example, if you are evaluating a program that has been implemented, the starting point should be on the ‘theory’ that was used by the program funder/designer. In a three year evaluation that we recently undertook, the client required us to undertake a systematic review to identify evidence that would support their program. This in itself raises issues such as, what if the evaluator finds that no single underlying theory of change exists?, what if the evidence is weak (which it invariably is) or mixed (which it is also often the case for some areas of social policy). Is it really up the evaluator to try to find the ‘right’ theory of change that can then be used to retrofit the program?

In this context, the starting point should be to go back to the funder/designer to find out their views as to why the program was expected to work, the underlying rationale and then for the evaluator to test the strength of the client’s theory. If the evaluator finds that there is a weak evidence base or worst still a contradictory evidence base, then this starts for an interesting conversation. Should the evaluator also be asking the questions regarding the quality of program development if there is no underlying theory of change or evidence base supporting the program? Was the program intended to test a specified theory of change or was it set up largely as a result of political factors?  What if these questions are uncomfortable?

If the context is different in that the evaluator is brought in early, then through the theory of change process, the evaluator is in a position to influence program design, based on the nature of the underlying  evidence and theory. Furthermore, it is possible to have explicit discussions about the factors other than evidence and theory that influence program design .  

The primary role of the evaluator should be on understanding the client’s theory of change and then testing whether the program has been implemented in a way that is consistent with this theory and ‘optimises the theory’ i.e. is it likely that the program will generate the expected benefits given what and how it has been implemented.”

 Additionally there is value in identifying the additional factors that influence program design and how these factors should be addressed in an evaluation. For example are these other factors  constraints on setting up a program that is intended to be consistent with evidence nad the theory of change? Alternatively are they the factors that define the program and is the evidence and theory of change being used selectively to post justify a program design.  These two possibilities have important implications for how an evaluation is conducted.

Anonymous's picture
Scott Bayley

An excellent resource, thank you Patricia. I have shared this with with staff in DFAT's aid program.

regards

Scott Bayley
Principal Specialist,
Performance Management and Results

mikkelml's picture
Mikkel Møldrup-Lakjer

John Pilla points to some interesting questions in situations where there is no explicit or agreed theory of change to guide the evaluation. As programmes may have more than one "author", it is not even always possible "to go to the source" in order to elicit the "original" theory of change. Predefined indicators are indeed often as much (or more) about political accountability - the expectations of other stakeholders - as they are shaped to be congruent with a theory of change.

Even if it is not always easy to make sense of programmes, I subscribe to the approach of "reconstructive interpretation", as proposed by Vedung and others. We should assume that there is a rational reasoning underlying or embedded in an intervention. The assumption is an analytical tool which can help us to extract a theory of change by making sense of the programme's components - even though no one may have thought it through that way before:

If a programme is supposed to bring about these results with these means - how could this possibly take place?

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.