Evaluating policy influence and advocacy
Influencing and informing policy is the main aim for many development organisations. However, activities directed at policy change are, in general, very hard to monitor and evaluate.
As policy change is often a complex process, it is difficult to isolate the impact of a particular intervention from the influence of other factors and various actors. In addition, monitoring and evaluation tools usually used in managing interventions can be difficult to implement in these contexts.
Policy influencing techniques and approaches
Robust evaluation of policy influence is far from unachievable, however. In fact there have been numerous contributions to the field that help us unpack this tricky subject. Start and Hovland provide us with a good starting point by proposing a useful heuristic to understand the variety of policy influencing strategies that exist. They identify four categories of organisations depending on the option of influencing, and the role of rational evidence versus value or interest-based argument used by an organisation. The following diagram demonstrates how to find your organisation in this typology.
The four categories of policy influencing techniques and approaches characterised by Start and Hovland can serve as a starting point to find advocacy evaluation tools suitable for your organisation. Below we provide summaries of each approach.
Academic research is most commonly evaluated using academic peer reviews and a number of citations in research publications. However, for evaluation of policy influencing activities these methods are clearly insufficient. As a lot of policy-oriented research institutions rely on donors or public funds, there is a growing need to demonstrate the actual influence of research projects on policy and practice.
Fred Carden, in a paper on assessing the policy influence of research, discusses the issues with assessing the influence of research on policy change and emphasises the importance of the context of the situation. He notices however that the excessive inclusion of context in the evaluation increases the difficulty in claiming the impact of a particular intervention. He advocates for use-oriented approaches, which engage the users of evaluation findings at all levels improving validity of the evaluation.
For further practical advice, there are two tool kits well worth browsing: this comprehensive handbook by CIPPEC is designed to improve the performance of research institutions through developing a system of monitoring and evaluation of the impact of their own activities on policy change; and Ingie Hovland's 2007 paper, Making a difference: M&E of policy research presents a number of evaluation methods divided into 5 categories: strategy and direction, management, outputs, uptake, and outcomes and impacts.
Annette Boaz et al present a thorough review of a number of popular methods used in evaluating the impact of research on policy (including ethnographic and quantitative approaches, focus groups, process tracing, and network mapping and analysis). They summarise the pros and cons of using qualitative and quantitative approaches and suggest that qualitative methods, such as semi-structured interviews, documentary analysis, field visits and observations are more suitable for the analysis of research impact on policy, but they also acknowledge the value of mixed-method approaches. They conclude with a very useful tool for helping design evaluations of this type, which involves 8 questions to consider:
- What conceptual understanding of the relationship between knowledge and policy is informing the evaluation?
- What are the outcomes of interest?
- What options might be used to explore the outcomes of interest?
- How does the evaluation address issues of attribution?
- What is the direction of travel for the evaluation?
- Is this a mixed option approach, providing scope for triangulation?
- Will the methods selected capture the context and complexity of the research utilisation pathways, therefore helping to understand how (and whether) change has occurred?
- Does the timing of the evaluation offer sufficient time for change to occur, without compromising the likely recall capacities of respondents?
Policy influencing based on building public support for a new policy relies on public messaging and campaigning in order to engage large numbers of individuals. Although this approach has been used by various groups for a long time, measuring the actual influence of advocacy activities remains problematic. This is especially the case with changing public attitudes and preferences as there are multiple factors affecting people’s choices and behavioural change.
Harry Jones proposes a range of methods that may be useful in assessing the impact of public campaigning. For example, surveys or focus groups may help to both measure and understand attitudes and preferences of a certain target group. It is also important to monitor the media as it may be crucial to explain behavioural change.
A good example of an advocacy evaluation is presented in a paper by Michael Quinn Patton. He discusses a case in which he assessed the influence of judicial advocacy efforts targeted at the Supreme Court. The intervention’s impact was evaluated using evidence gathered through fieldwork (interviews, document analysis, detailed review of the Court arguments and decision, news analysis, and the documentation of the campaign itself), aiming at eliminating alternative or rival explanations until the most valid explanation remained (i.e. using forensic method or “modus operandi” approach).
In response to the difficulties with operationalising and measuring advocacy efforts, Coffman and Reed, in their paper 'Unique Methods in Advocacy Evaluation', describe four newly developed methods created specifically for assessing advocacy and policy change. These are Bellwether Methodology, Policymaker Ratings, Intense Period Debriefs, and System Mapping. The Innovation Network has also published a case study example of system mapping used in an evaluation of the advocacy efforts of an international aid and relief organisation.
Lobbying is generally believed to be very hard to capture and analyse in a standardised way as it relies on informal interaction and takes place in highly fluid contexts. However, Harry Jones lists some tools that can be used to evaluate the contribution of lobbying in policy change, including recording observations from meetings and negotiations, interviewing informants, and conducting qualitative, in-depth analysis of different aspects of lobbying activities.
In addition, dividing lobbying into smaller components may help to evaluate it more effectively. For example, Start and Hovland divide lobbying into three levels: Need to Know, Need to Inform, and Need to Negotiate. They also provide some useful tips for lobbyists, emphasizing the importance of planning a strategy, preparation, building relations and give suggestions on how to handle the lobbying outcomes.
The Centre for Lobbying in the Public Interest helps to improve the advocacy impact of non-profit organisations. They provide a short guide to evaluation of lobbyists work. They advise to evaluate the lobbyist work, skills and attributes using following categories: the ability to build relationships, perseverance, organizing the grassroots, coalition building, ability to motivate and communicate with various target groups, proficiency in the use of communication technologies, knowledge of the basics about the legislative process, and the organizational structure.
Activism aims at achieving change in policy through pressure. Therefore it usually uses confrontation as the method of advocacy strategy and works from outside of policy communities. Activism is an important part of obtaining policy change, yet there is very little academic work dedicated to the analysis of the tools and methods used by activist organisations. Having said that, there are some online resources providing practical tips for nongovernmental organisations interested in pursuing direct advocacy.
Green Media Toolshed is committed to providing tools and improving the effectiveness of communications among environmental groups and the public. Their resources contain a collection of campaign manuals, handbooks, and planning tools targeted at civil society in the majority world.
In this article the authors argue that standardised approaches to monitoring and evaluation of policy influence might misguide advocacy institutions, as they don’t take into account how complex is the process of policy change. They also emphasise the need for greater cooperation among NGOs since it is very difficult to assess the influence of one organisation in isolation from others advocating for the same issue.
'Evaluating policy influence and advocacy' is referenced in: