Week 21: Know your measures - picking outcomes to monitor policy change
Simon Hearn continues BetterEvaluation’s theme on the monitoring and evaluation of policy change by suggesting a set of measures to help those struggling to monitor the slippery area of policy influence and advocacy.
Policy change is one of those slippery notions that we often struggle to grasp– what does it actually mean and how can we measure it? Many programme designers recognise that to bring about sustainable development requires action at multiple levels: for example, as well as building water infrastructure and building the capacity of service providers we also need to assess the legislative and regulatory environment that constrains those water providers and perhaps generate evidence to inform better decision-making.
So if our programming is taking on these new dimensions, is our monitoring and evaluation keeping up? Are we able to say as much about the efficacy and effectiveness of policy influencing interventions as we can about infrastructure development or capacity development interventions, for example?
The first thing to decide is what type of outcome we can look for that will tell us whether our interventions are having the desired effect. Largely this will depend on our strategy, but we can define some archetypes to choose from. Much of the policy change we are looking for is formal changes in legislation, budgets or programmes: but there are many important changes which come before and after these to policy change and ensure it is effectively implemented or enforced.
For several years, in the RAPID programme at ODI, we have adopted a model of stages of impact on policy that was first developed in the late nineties by Margaret Keck and Kathryn Sikkink in their book on transnational policy advocacy (used by Jones and Sumner, 2011). They talk about five stages of policy change:
1. Issue creation and agenda setting
2. Discursive positions of state and organisations
3. Institutional procedures
4. Concrete policy change
5. Behaviour change in implementation.
These five make a great start in understanding some of the things we can measure to know if things are moving in the right direction but there are others which focus on different actors in the process. Another useful typology comes from policy analyst, David Steven, who talks about five intermediate policy outcomes (used by Batchelor, 2012) which overlaps with Keck and Sikkink’s but includes two more: the development of networks and coalitions and developing capacity of organisations to engage.
For an upcoming publication (a guide to the RAPID Outcome Mapping Approach, which helps organisations and researchers diagnose a policy problem, make a plan to influence policy and monitor and learn from their activities) I merged these into a new list and also threw in a measure on public opinion which is important for a lot of influencers:
- Attitudes of key stakeholders to get issues onto the agenda: How interested and open are policy actors to your issues? What kind of evidence will convince them?
- Public opinion: How are the public engaged in these issues? How are they informed or motivated?
- Capacity and engagement of other actors: Who else is engaging in this policy area? How influential are they? What can be done to involve others?
- Change in discourse among policy actors and commentators: What are the influential policy actors saying on this issue? How is their language changing?
- Improvements in policy-making procedure/process: Who is consulted during policy-making? How is evidence taken into account?
- Change (or no change) in policy content: What new legislation, budgets, programmes or strategies are being developed?
- Behaviour change for effective implementation: Who is involved in implementing targeted policies? Do they have the skills, relationships, incentives to deliver?
- Networks and systems for supporting delivery: Are different actors working coherently together to implement policy? Are the necessary structures and incentives in place to facilitate this?
Considering our results around these eight areas will help us to build a holistic picture of our effects across the system in which we are engaging. It will help us to keep our eyes on the prize but also to focus on the smaller, incremental changes that are fundamental for sustainable change. It also helps us identify informal (e.g. 1 and 4) changes as well as the formal (e.g. 5, 6 and 7); indirect (e.g. 2, 3 and 8) as well as direct influence (4, 5 and 6); and change at the level of systems (e.g. 3 and 8) as well as individuals (1, 4 and 7).
Not all of these will be relevant for every policy influencing intervention – in fact it will be very rare that all of these measures will be prioritised in an M&E framework because they apply to different theories of change. For instance, public opinion will only be important for interventions that rely on mobilising the public through mass media to exert pressure for political change. Likewise capacity and engagement of other actors will only be important to measure if your strategy relies on indirect influence, through for example, developing the enabling environment for civil society to work more equitably with parliamentarians.
A typology like this is only the first step to tacking the challenge of measuring policy change – the next step, after deciding the outcome measures, is designing the data collection instruments and planning how to understand the causes of the outcomes. I leave you with some good examples of where I think this has been done well:
- Supporting international climate negotiators: a monitoring and evaluation framework, Dan Hamza-Goodacre, Stuart Jefford and Nigel Simister.
- DFID Influencing in the Health Sector, Jeremy Clarke, Enrique Mendizabal, Henri Leturque, Veronica Walford and Mark Pearson.
- Evaluation of ODI’s EDCSP Project, Sarah Bayne.
- Informal Traders Lock Horns with the Formal Milk Industry: The role of research in pro-poor dairy policy shift in Kenya. Cokro Leksmono, John Young, Nick Hooton, H. G. Muriuki, & Dannie Romney.
Do have experience of working in or monitoring advocacy or policy influence programmes? Do you think there are any outcomes that Simon has missed?