Adapting evaluation in the time of COVID-19 — Part 2: Define

Adaptive%20Evaluation%20-%20Define%20blog.png

The Covid-19 pandemic has led to rapid changes in the activities and goals of many organisations, whether these relate to addressing direct health impacts, the consequential economic and social impacts or to the need to change the way things are done.

Evaluation needs to support organisations to use evidence to plan these changes, to implement them effectively, and to understand whether or how they work – in short to articulate an appropriate theory of change and use it well. 

In the second of our Adapting Evaluation in the time of COVID-19 blog series, we are focusing on the second cluster of tasks in the Rainbow Framework – DEFINE what is to be evaluated. To do this, we’ve brought together some advice and resources on how to more effectively develop, represent and use theories of change in the current context – whether you are reviewing the theory of change for an existing initiative or developing a theory of change for a new initiative.

1. Take the time to develop or revise the theory of change

In such difficult times, it’s worth checking that it’s appropriate to put time and focus into evaluation activities, including developing or revising a theory of change.  Is it just a luxury that we can’t afford at the moment?  We don’t think so. As Carol Weiss and Ray Pawson, pioneers in program theory and realist program theory respectively, have noted, quoting Kurt Lewin, “there’s nothing as practical as a good theory”. 

Program theory can be used to improve planning before an intervention begins, helping to identify gaps or contradictions that would undermine successful implementation or the achievement of intended outcomes. It can be used to communicate with stakeholders so they all understand what is most important.  It can be used to guide the design of monitoring and evaluation to support effective implementation and learning for future efforts.

In a recent blog, Michael Patton urged evaluators and those working in evaluation to actively support the changes needed in response to the COVID-19 pandemic:

Embrace change, don't resist it. Program goals may appropriately change… Target populations may change. Implementation protocols may change. Outcome measures may change… Expect change. Facilitate change. Document changes and their implications. That’s your job in a crisis, not to go on in a comfortable business-as-usual mindset. There is no business-as-usual now. And if you don’t see program adaptation, consider pushing for it by presenting options and introducing scenario thinking at a program level. Take risks, as appropriate, in dealing with and helping others deal with what’s unfolding.

It is quite likely that changes will need to be made to existing theories of change. And that theories of change for new interventions will also need to be reviewed and changed as the situation continues to change. A theory of change should not be a rigid tool that inhibits change but instead should facilitate making and communicating appropriate change.

2. Have an actual theory in your theory of change

Too many so-called theories of change are just a sequence of inputs, activities, outputs and outcomes with a non-theory statement that IF we run these programs THEN we will achieve these outcomes.  This does not explain what it is about the programs that is understood to lead to the outcomes — if you don’t have a theory that explains how it works, it’s not possible to know how to adapt it to meet the changing situation.

What’s behind the change you expect to see?

For example, in a hand hygiene program, what is it that leads to changes in hand washing behaviours?

  • If it’s mostly about providing information, does it work by filling in known gaps in information (e.g. answering questions about whether soap is adequate or hand sanitiser is needed), or by supporting people to work out how to apply information in their particular situation (how to set up handwashing stations in their home or workplace),  or by raising new issues previously they’d been unaware of (e.g. the importance of washing hands for 20 seconds or more)? 
  • Or does it work quite differently — for example, by providing aspirational models of behaviour, by changing social norms, or by triggering reminders, or providing logistical support? Or all of these? 
  • Or does it work in one way for some people and in another way for others?

As many people have long argued (going back at least to Edward Suchman in 1967 and Carol Weiss in 1972), a good theory of change has one or more explicit change theories – how change is understood to come about,  and for each of these some explicit action theories — how the program activities will trigger these change theories.

What adjustments do you need to make? 

If you are revising the theory of change, it is useful to be clear about the level of change that is required.

  • Are you just adapting your activities (action theory) or fundamentally changing how the intervention will work or what it is trying to achieve (change theory)?

Does the program design, or the adjustments to the program design, support the way you expect change to happen?

Having explicit change theories in your theory of change makes it easier to do a check of the intervention design before implementation.

  • Has the intervention been designed in a way that will support the important change theories — for example, providing opportunities for informal discussion between participants who live near each other to encourage them to share information about local resources for handwashing or to strengthen social norms? 

If the program delivery has to change in the current context, a good theory of change can be used to evaluate options to make sure they still support the important change theories.

  • For example, sending information by email when face to face meetings are not possible might fill knowledge gaps but what else would need to be done to influence social norms?
  • If the delivery has moved to videoconferencing, how can this be done in a way that does not simply involve one-way sharing of knowledge but also supports sharing knowledge of local adaptations?

A good theory can tell us what to really pay attention to, what to retain when we are having to leave out some components of the intervention, how to adapt in ways that keep the essence, and how to apply what has been learned in different contexts.

Blog — ​Having a theory in the theory of change: This BetterEvaluation blog discusses why it’s important to have a theory of HOW you expect activities to contribute to the intended results.

3.Taking a systems approach to develop your theory of change will likely be useful in the current circumstances

The ripple effects of the pandemic around the world and across sectors and organisations have made evident the importance of seeing our interventions not as a closed system, with tightly controlled inputs, processes, outputs and outcomes, but as an intervention that is part of much larger systems. 

It is time to draw on the long experience of using systems approaches in evaluation. A systems approach involves thinking carefully about three things — boundaries, interactions and perspectives, and the consequences of the choices made in relation to each of these. 

Boundaries

Think about the boundaries of what is included in the theory of change.  All initiatives take place in a larger geographic, political and social context. 

  • How much of this needs to be included — either because the context is an important influence on the initiative or because the initiative seeks to influence the context (or both)? 
  • Is the intent of the intervention to make small changes within the existing system or to transform the system? 
  • If the latter, do you need a theory of transformation?
  • How can you include consideration of potential negative outcomes, and how these might be avoided, ameliorated or at least identified early?

Interactions

Think about the different interactions involved in the theory of change and whether change will be non-linear. 

Diagram of different loop types

This can include virtuous circles, where initial early positive outcomes reinforce the mechanisms of change — for example, wide and visible uptake of new behaviours changes people’s perceptions of social norms and encourages more to change their behaviour. 

It can include vicious circles, where initial negative outcomes trigger negative causal processes — for example, unsuccessful attempts at a new practice can lead to decreased motivation and effort, which leads to even less successful efforts. 

It can also include balancing loops, such as stabilising or stagnating where early success undermines the processes that led to that success — for example, in a stagnating loop, if social distancing strategies lead to reduced incidence of disease, and less perceived risk, this can lead to lack of compliance and increased incidence. 

There can also be tipping points where a small change can make a big difference once a threshold level has been reached - for example herd immunity or a shift in social norms.

These loops are illustrated in the image, taken from The Omidyar Group workbook, Systems Practice.

  • Are there likely to be feedback loops (vicious circles, virtuous circles, balancing loops)?  Tipping points?

Perspectives

Think about the different perspectives involved in the initiative.

  • How can you ensure these are understood and included in the process of developing the theory of change?
  • Is there agreement among different stakeholders about the intended outcomes or how they might be achieved? 
  • Can these different views be accommodated within one theory of change or are they mutually exclusive? Which perspectives should be prioritised?

These issues will affect who should be involved in the process of developing the theory of change and how.

Resources:

Option page — ​Negative programme theory: identifying ways in which program activities might produce negative impacts rather than their intended impacts.

Website — ​Systems in Evaluation TIG (SETIG): SETIG is an online community of practice for members of the Systems TIG of the American Evaluation Association and others interested in systems thinking for evaluation practice.

Discussion paper — ​Principles for effective Use of Systems Thinking in Evaluation - SETIG: This set of principles was created by SETIG to guide the use of four core systems concepts in the design and implementation of an evaluation. The primary purpose of these principles is to support evaluators and evaluation stakeholders in the use of systems concepts in evaluation.

Blog — ​The Global Alliance for the Future of Food Makes History with Formal Adoption of a Theory of Transformation: Michael Quinn Patton discusses the shift from ‘theory of change’ to ‘theory of transformation’, in the context of the Global Alliance for the Future of Food’s ground-breaking adoption of an official theory of transformation, which incorporates multiple theories of change at multiple levels to achieve local and global action and interaction for transformational change in collaboration with other committed stakeholders.

Blog — ​How systems mapping can help you build a better theory of change: Chris Alford provides examples of using feedback loops, and using theories of change to inform single loop learning (error detection and correction) and double loop learning (reviewing and revising assumptions).

Guide — ​Systems Practice: This workbook from the Omidyar Group provides detailed guidance to people working on complex problems around social change who want to explore ways of using non-linear causal mapping to better understand, plan and communicate their theory of change.

4. Draw on a range of sources to develop your theory of change

A good process for developing a theory of change draws on a range of evidence, including evidence from previous research and evaluation, tacit knowledge and practice wisdom, lived experience from intended beneficiaries and key informants.  It can also draw on formal change theories.

Think about the process you will use to develop or revise the theory of change.

  • How will you find relevant and credible evidence to inform your theory of change?
  • How will you engage with stakeholders and people with useful knowledge and experiences? This includes marginalised groups of stakeholders with whom engagement may be more difficult in current circumstances, but no less important.

Holding a workshop to develop a theory of change virtually will require different thinking, planning and processes to holding this type of workshop face to face. We’ve included some examples of doing this below, including from Barbara Klugman who sums up the process nicely:

“I think doing serious thinking work in a way that strengthens group cohesion around what they are doing and why they are doing it, requires attention to the immediate context, the emotional space participants are in, and the ensuring of individual space for reflection and generating of new ideas alone and drawing on collective wisdom. Doing this using an online platform, and with people from multiple time zones, means slowing the pace, listening carefully, and shifting plans as needed.”

Resources:

Examples:

Coping with Covid-19: Theory of Change workshop online: This blog post on the Gender and Evaluation website by Barbara Klugman describes the lessons learned from adapting a two-day TOC workshop originally planned for a face to face meeting into a virtual workshop.

Theory of Change Workshop Online – pre-COVID-19: Also on the Gender and Evaluation website, as a follow up to Barbara’s post above, this blog by Michaela Raab describes the process of facilitating a theory of change workshop online before the COVID-19 restrictions. It includes a practical description of what they did, along with helpful suggestions for what could have been done better. The discussion of the comments also contains some good suggestions for technology, as well as useful guidance on how make online meetings and webinars more accessible for people with disability.

How to Make Your Virtual Meetings and Events Accessible to the Disability Community: This guide outlines a number of ways to make your virtual meetings and events more accessible.

Resource list — ​Theory of change software: including options that aid online collaboration

Task page — ​Develop theory of change: Our task page in the Rainbow Framework has more information and options for developing a theory of change using different sources.

Blog — ​BetterEvaluation FAQ: How do I choose a suitable theory of change?: This BetterEvaluation blog, in response to a question from our community, discusses using different methods to develop a theory of change, including using ‘big t’ theories, policy documents and background papers, stakeholder mental models, and logical analysis of causal pathways.

Blog — ​A quick primer on running online events and meetings: This BetterEvaluation blog gathers advice and resources for moving your face-to-face interactions online.

5. Use theory of change to help navigate uncertainty

It’s not always possible to have the necessary information to be able to plan an intervention in detail in advance.  There are gaps in information (known unknowns) and gaps in our understanding of what we need to know (unknown unknowns), and as the situation changes rapidly our priorities, our resources, and what will work is changing too.

It’s helpful to remember that a theory of change is not like instructions for making a packet cake mix, or assembling a jigsaw, where everything is known and planned in advance.  It’s a way to help us think, communicate, make sense of information, and act.

It can therefore be a useful tool in conditions of complexity and uncertainty.

The area of adaptive management, especially in international development, provides some ways of using theories of change under conditions of uncertainty.  This involves developing light versions of theories of change which include other actors involved in the change, and which are designed to be used, reviewed and revised during implementation.  These are used to guide small actions and to learn from them and to gradually develop a better understanding of how things work in this particular context and how to act effectively.

For example, Richard Allen has described how his organisation has used theories of change for complex interventions involving developing an understanding of the system that it intends to address and how change occurs, then layered theories of change and specific action theories of interventions, see image.

The Asia Foundation developed an approach called Strategy Testing to help navigate and monitor complex and fluid development problems through an iterative, adaptive approach. The first step in Strategy Testing is developing an initial Theory of Change, which “sets out the team’s ‘best guess’ about the most likely path to change. Since this first TOC is based on the team’s initial understanding of the problem and its context, they recognize that the TOC is likely incomplete and will evolve over time as the team builds relationships, gathers new information, experiments, and, most importantly, reflects on what is working and what is not” (Ladner, 2015, p.10).

Resources:

Blog — ​Acting in complex times: In a recent blog, Cameron Norman discusses ways of starting with small explorations and efforts at coherence and building up to understanding larger systems.

Resource — ​Strategy Testing: This discussion paper by Debra Ladner for the Asia Foundation lays out the Strategy Testing approach, which requires teams to continually assess the probability of achieving success as they process new information and refine their strategies.

6. Represent theories of change appropriately 

How we represent our theories of change makes a big difference to their utility.  Simple linear boxes of inputs-processes-outputs-outcomes and impacts have rarely been adequate and are even less likely to be so given the issues discussed above.

In a review of theories of change diagramming conducted as part of CEDIL (Centre for Excellence in Development and Learning), Rick Davies identified the following common problems:

  1. Unlabelled connections which don’t indicate how this causal connection works, or anything about timing, duration or scale
  2. Missing connections - where the theory of change does not show how the various components are connected
  3. Symmetric connections which show a false equivalence of the importance of different components
  4. Lack of clarity on multiple causal pathways – where the theory of change does not make it clear whether these work as alternatives or as a necessary combination, and if so whether these are sequenced or simultaneous
  5. Lack of adequate attention to important feedback loops

The changes we make to more usefully represent our theories of change might involve some simple additions, such as:

  • using outcome hierarchies rather than pipeline models to make it easier to show sequencing of activities and the contributions of other partners and external factors
  • adding details in a narrative, table or on a diagram about the nature of the connections
  • adding important feedback loops

Or it might involve quite different forms of representation, including expressing it in terms of principles or using network diagrams as part of it.

Resources:

Discussion paper — ​Representing Theories of Change: A Technical Challenge with Evaluation Consequences Rick Davies paper on problems with drawing logic models This paper sets out some common problems with theory of change representations and some suggestions in terms of improving the processes of developing them.

Blog — ​BetterEvaluation FAQ: How do I use program theory to evaluate a system? This blog explores various ways theory of change diagrams can be used to evaluate systems, and some different ways of representing these.

Conclusion

If you’ve not found theories of change useful in the past, or you don’t think they are worth the time or effort in the current crisis, we encourage you to take the time to think through ways of developing and using them that could be more useful to your program and evaluation efforts. As Craig Valters recommended, in his paper on a radical approach to learning in development, a theory of change is most useful when thought of as a compass rather than as a map. Taking a compass with you when navigating the uncharted waters of COVID-19 seems like a pretty good idea.

 

'Adapting evaluation in the time of COVID-19 — Part 2: Define' is referenced in: