52 weeks of BetterEvaluation: Week 12: Having an adequate theory of change

21st March 2013 by Patricia Rogers

Many evaluations use a theory of change approach, which identifies how activities are understood to contribute to a series of outcomes and impacts. These can help guide data collection, analysis and reporting.

But what if the theory of change is has gaps, leaves out important things – or is just plain wrong?

Risks when the theory of change is inadequate

Projects and programs that are based on an inadequate theory of change are less likely to be effective as plans and activities will not cover everything that needs to be done, and projects will be implemented when there is little chance of success. Evaluability assessment includes assessing the quality of the theory of change before implementation, while changes can be more readily made.

Evaluations that are based on an inadequate theory of change can focus on the wrong things.

For example, think about an evaluation of a health promotion project, such as encouraging safer sex practices to reduce HIV/AIDS transmission.An inadequate theory of change might only focus on providing information in order to change behaviour, an inadequate theory that leaves out many important other aspects needed to initiate and maintain behaviour change - such as social norms, aspirations, skills to practice certain behaviours, and access to required materials, such as condoms.

An evaluation that is based on this inadequate theory of change might then only collect data on the identified intermediate outcomes of knowledge change and behaviour change.

If the project is not working, an interim evaluation might find that there have been improvements in knowledge, and conclude that the program is on track.

If the project is working, because it includes or links to other activities that address these broader issues, an end of project evaluation would not adequately show what is needed to replicate this success in other projects.

How to check the quality of a theory of change

1. Make sure you're solving the right problem.

For example, a health promotion project needs to understand why people are not practicing safer sex, and whether knowledge is an important part of this.

While this situation analysis should have been done as part of project design, it might not have been done adequately - or the situation might have changed since then.

What are the needs that the intervention should address?
What is the problem the intervention should solve? What causes this problem?
What are the strengths the intervention should build on?
What are the opportunities the intervention should capitalize on?
Who else is addressing this situation? Are they potential allies or opponents?
How well understood is the situation?
How changeable is the situation?

These questions, from the book Purposeful Program Theory can be used to develop a situation analysis.

2. Use a diverse range of sources, including critics and intended beneficiaries

Having a range of perspectives on the project, and on the emerging theory of change, can help to identify and correct inadequacies. People who have been critical of a project can be especially helpful in identifying gaps and unlikely links.

When looking for gaps, remember that a theory of change can't include everything - the map is not the territory. So focus on making sure it includes all the important things. A good way to do this is to ask people "What is it about this program that makes the difference?".

3. Check outliers and exceptions against the initial theory of change

If you have data about the project already, identify people or sites that don't fit the pattern and see how the theory of change can modified to include these examples. Don’t only focus on the average effect of the program, especially if it work very differently in different implementation environments or for different types of people.

4. Check if there are complicated and complex aspects that need to be addressed

A simple theory of change might be inadequate if it does not address important aspects such as:

Multiple and conflicting intended outcomes
Long causal chains involving multiple actors
Different causal mechanisms operating in different contexts
Recursive causality, involving feedback loops
Emergent outcomes

(For suggestions on how to represent these in logic models, see examples in this article Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions).

This post is the second in a series of three from a workshop held at the Community of Evaluators Evaluation Conclave, Kathmandu, where we shared some ideas on avoiding three common mistakes in evaluation.

Do you have other strategies to suggest? Examples to share? Or other common mistakes to discuss? Register and login to add comments.

You’ll find more information about different ways to develop and represent a theory of change on our task page.