BetterEvaluation FAQ: How do you use program theory for evaluating systems?

Patricia Rogers's picture 21st June 2017 by Patricia Rogers

In our recent blog post about using theories of change and logic models better in evaluation, we asked BetterEvaluation members to submit a question or challenge that they have in relation to creating or using theory of change for review by the BetterEvaluation team. One member asked us for advice on using a theory of change for systems change which we are very happy to share below:

Although it’s sometimes referred to as program theory or program logic, theories of change can be used for interventions at any scale, including policies, whole-of-government initiatives, and systems.  Evaluating at this scale presents particular challenges and scope for using different tools and techniques.

In answering this question, I think it’s useful to start by considering whether it’s helpful to think about the system as simple, complicated and complex – in particular distinguishing between what is complicated (lots of components that can be identified in advance) and complex (emergent and adaptive components, actors, goals and roles).  (You can find more information about this distinction on our thematic page on complexity.) It might well be the case that it’s most useful to think of some aspects as complicated and some as complex.

Thinking about it as a simple system

One way of using theories of change for evaluating systems is to treat them the same as a project and use a very simple version of a logic model: inputs -> activities -> outputs -> outcomes -> impacts. 

In the early days of the American Evaluation Association Topical Interest Group for Systems in Evaluation, we used to get a number of proposals that were about evaluating a health system, or an education system, or a justice system, but which really address the fact it was a system and not a simple project in how it was evaluated.

For example, if you were evaluating a new professional development intervention for teachers that is intended to improve the school system, especially in terms of reducing school drop-out rates, you could do simple logic model something like this:

The problem with this is that it doesn’t really acknowledge and address any systemic issues.  While it hints at two systems (the education system that is focused on teaching and learning, and the welfare system that is partly based in schools and partly in community agencies) it doesn’t focus attention on how these need to be better connected and how they might need to exchange information and be responsive to each other.

I wouldn’t usually recommend using this type of logic model for evaluating a system, because it leaves so much out.  But there are some situations where this might be an important starting point.  For example, several years ago when I was working with the Public Service Commission of South Africa, we developed the following diagram to support their work contributing to improving the public service monitoring and evaluation system.  Because this had such a large scope, we developed a very simple outcomes hierarchy, focusing on a few key issues, which could provide a common road map for discussions and planning (you can read more on this in the November 2004 PSC Newsletter, page 4). 

This outcomes hierarchy was intended to focus attention on the behaviour change theories underpinning performance monitoring.  They might partly work by identifying problems that can be identified (and many of the monitoring systems focused on this), but they also should aim to work by providing clear signals about what is valued, and by recognising and sharing information about successes – that could then be further investigated to find out how they were achieved.  They also cannot work if they only involve reporting data – some process for follow up or response is needed.

This simple theory of change could be then used to plan, manage and evaluate a departmental monitoring system or a system across government.   

Identifying and addressing complicated aspects

It’s often much more useful to think about the system as complicated – having lots of components and actors which can be identified in advance and whose contribution to the theory of change can be articulated. There are two main ways of doing this – hierarchical and parallel.

Going back to our school drop out example, a hierarchical theory of change would think through the connections between the classroom, the school and the school district or state.  For example, in order for teachers to undertake additional activities with students, the school might need to provide additional resources or change policies or processes; in order for schools to be able to do this, school districts or states might need to also provide resources or change policies or processes.

A parallel theory of change would show the educational system needs to connect better with the child welfare system:

A few years ago, I contributed to the development of a monitoring and evaluation framework for a whole of government initiative around family violence which connected frontline services, housing, police, and courts.  At the heart of the framework was a parallel version of an outcomes hierarchy which showed how different service systems contributed to achieving particular outcomes and therefore needed to be connected.

The following example from the National Health Service in Scotland uses a parallel pipeline logic model, showing how the outputs from different components contribute to producing overall outcomes of reducing tobacco smoking.

(National Health Service, Scotland - Simple Guide 2: Results Chains and Logic Modelling, p6)

Treating it as a complex system

Complicated approaches work when the components and how they fit together can be articulated in advance.  But sometimes it is not clear how a system should be defined (what is in and what is out), how it should change and who should be involved in doing this. 

Some approaches to systems change do not yet have an explicit theory of change but have identified good practices that speak to an underlying implicit theory of change around connecting different perspectives and information.

For example, in a recent article in the Stanford Social Innovation Review, Mark Kramer discussed five effective practices being used by philanthropic organisations working to bring about systems change:

  • First, these funders work both inside and outside the nonprofit sector, leveraging the market forces that drive for-profit companies and making efforts to improve the implementation and outcomes of existing government programs.
  • Second, they build common ground among key actors by forging cross-sector coalitions.
  • Third, they recognize the importance of the intangible narrative that underlies the public response to our society’s problems and work actively to change that narrative.
  • Fourth, they elevate the voice of lived experience in shaping solutions.
  • Fifth, they reconsider their own staffing, budgeting, and operations to address racial and cultural blind spots, focus on multifaceted problems rather than separate program areas, and develop more active leadership roles for their CEOs and boards.
in the Stanford Social Innovation Review (Apr 11, 2017)

Julia Coffman has identified elements that are needed in a theory of change for building systems (such as an early childhood system that brings together early learning, family support, health, mental health and nutrition and special needs/early intervention):

Another way of thinking about theories of change for systems change is to use specific theories of systems change.  For example, the Berkana Institute Theory of Change draws on the two loops model which shows how the old system might change to a new system:

As one system culminates and starts to collapse, isolated alternatives slowly begin to arise and give way to the new. 

In this video Deborah Frieze, Berkana’s former co-president, explains the two loops theory and speaks about the way that our work to name, connect, nourish and illuminate has fit into this model. She also identifies some of the different roles we might play to hospice the dying system, usher in the alternative system and make clear the choice between the two.

The Berkana Institute, Our Theory of Change

An example by Catherine Howe of a theory of change for systems change that’s expressed in the form of three principles:

1. Map your system levers and maintain this as a dynamic map 

2. Cultivate and curate a network of active participants who want to work together differently 

3. Create some kind of engine of change which can be brought to bear on system-wide problems which will further develop the system change map as well as the network of active participant

in Curious Catherine Blog

Catherine's diagram of a principles-based theory of change such as this looks very different to the classic pipeline model or an outcomes hierarchy: 

Some points to consider when using program theory for a system

What are considered the boundaries of the system?  Are there different views on this?  What might be the implications of drawing the boundaries tighter or wider?
Is it most useful to think about the system overall as simple, complicated, or complex?  Are there some aspects that would be useful to think of differently (for example, a simple aspect of a complex system, or a complex aspect of a complicated system)? 

What do you think?

Are these options and examples for theories of change for systems change helpful?  Do you have other techniques and examples to share? We'd love to see some of your own ways of representing a theory of change for systems change - email us or Contribute Content via the website.

[UPDATE: 11:00am 23.06.17 AEST)
 
Thank you to those who've been in touch with us to share your thoughts or questions on this blog. In response, we'd like to add some resources for those who are interested in reading more about systemic thinking in evaluation.

Resources

Thinking systemically: Seeing from simple to complex in impact evaluation

This slide show is from a presentation given by Professor Patricia Rogers, Dr. Irene Guijt and Bob Williams at the 2009 AfREA Conference. The presentation focuses on a using a systems approach to conducting impact evaluations. The presentation focuses on a using a systems approach to conducting impact evaluations. 

Prosaic or Profound? The Adoption of Systems Ideas by Impact Evaluation

This article, written by Bob Williams for the IDS Bulletin (Volume 46 Number 1), argues that in order to address the vulnerabilities of impact evaluation due to their complexity,  that it is essential to achieve a balance between the three elements of the systems field: understanding interrelationships; engaging with multiple perspectives; and reflecting on where boundaries are drawn in terms of those interrelationships and perspectives.

 

A special thanks to this page's contributors
Author
Director of BetterEvaluation/ Professor of Public Sector Evaluation, Australia and New Zealand School of Government.
Melbourne.

Comments

Ludwig Grillich's picture
Ludwig Grillich

Thank you for this post! 

Could you refer to any good practise examples where a initial programme theory was developed for a complex prevention program where impact comes trough cascaded activities?

Background:

We are actually developing a proposal for a formative evaluation of a program that trains multipliers who advice and train preventive specialists in companies so that they take action to reduce the health risks of employees in their company.

The main question of the formative evaluation is: how can the program produce the desired change in companies?

We decided to use the "Success Case Method" (Brinkerhoff, 2005) for answering the question. Never the less as a first step we will develop a program theory together with the steering committee and multipliers who will advise and train preventive specialists. We will do this in focus groups (specifying outcomes) and workshops (identifying causal links and core elements of the program)

We did similar several times for less complex interventions using the Swiss Model for Outcome Classification in Health Promotion and Prevention (SMOC) (Spencer et al., 2008).

Nevertheless SMOC doesn't´t fit for the cascaded activities in this program und doesn´t take into account any context. So we are looking for alternatives. Would be great if you have any hints!

Best regards,

Ludwig Grillich

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.