RAPID Outcome Assessment (ROA) is a methodology to assess and map the contribution of a project’s actions on a particular change in policy or the policy environment. It is a flexible and visual tool that can be used in conjunction with other evaluation tools and options.
ROA draws significantly from the Outcome Mapping approach as it focuses on key actors that the project is directly influencing and the progressive changes in those actors. It also draws from other methodologies and approaches such as Episode Studies, which focuses on working backwards from a policy change to determine the factors that contributed to it; and Most Significant Change, which helps to identify and prioritise the key changes.
The ROA methodology has three main stages. The first stage is a preparation stage, during which a document review and a series of informal conversations are carried out to develop a draft picture of the project’s history and the intended changes. The second stage is the workshop during which the key policy change processes are identified by the stakeholders. The third stage involves a follow up process that allows the researchers to refine the stories of change, identifying key policy actors, events and their contribution to change.
Stage 1: Background research and preparation
Stage 2: The ROA workshop
Stage 3: Triangulate and refine conclusions
- A set of episode studies tracked back from identified changes in the dairy policy environment in Kenya during and after the implementation of the SDP, and analysed the catalysts and preconditions which led to the changes
- A case studies tracked forward from the initiation of the SDP, highlighting key events in the project's history and identifying the immediate effects of its activities
- An outcome mapping exercise analysed the changes in behaviour of the key actors involved: government departments and parastatials, research organisations, private sector and CSOs
The diagram below is the output from the stakeholder workshop and describes the outcomes observed and the contribution of various factors to these. The horizontal axis is time, with the far left column representing the time when the SDP project began and the far right column representing the time of the workshop. The very bottom row represents the project, with key milestones written on coloured squared. The row above that represents the external environment, with note-worthy events written on squares. Each of the other rows represents the different actors involved in the process, with key observed behaviour changes recorded on squares.
This data was collected using a variety of options: document analysis, key-informant interviews and field visits as well as focus group discussions at the workshop. The main aim of the workshop, however, was to draw on the collective experience and knowledge of the participants to analyse the data and develop causal inferences between the different events and the outcomes. These are illustrated by the coloured lines: blue indicates direct influence, red is indirect influence and green is external influence. Each of the lines is numbered to link it to a narrative description of the influence, the evidence for the connection and the discussion behind the placement of the line.
(Source: Leksmono et al (2006). Informal Traders Lock Horns with the Formal Milk Industry: The role of research in pro-poor dairy policy shift in Kenya. Overseas Development Institute http://www.odi.org.uk/resources/download/135.pdf)
Advice for CHOOSING this option (tips and traps)
- This option helps address the attribution problem by mapping the effect of multiple factors and actors on observed or measured outcomes.
- This option requires experience with data collection (e.g. interviews, surveys, focus groups), analysis (e.g. episode studies, case studies) and facilitation skills.
Advice for USING this option (tips and traps)
- This option requires involvement of key informants throughout the study. These should include representatives of key actors, project implementers and other experts with knowledge of the field.
The data from interviews and focus group discussions will almost certainly need to be triangulated, clarified and independently verified if it is to be used as evidence.
- Making a difference: M&E of policy research.- The paper presents examples and approaches on conducting M&E of policy research from the current experience of a range of research institutes, think tanks and funding bodies.
- Informal Traders Lock Horns with the Formal Milk Industry: The role of research in pro-poor dairy policy shift in Kenya. - The paper presents the results of the analysis of policy changes that occurred during and after the implementation of the Smallholder Dairy Project (SDP) in Kenya
Hovland (2007). Making a difference: M&E of policy research. Overseas Development Institute. http://www.odi.org.uk/resources/download/281.pdf
Leksmono et al (2006). Informal Traders Lock Horns with the Formal Milk Industry: The role of research in pro-poor dairy policy shift in Kenya. Overseas Development Institute http://www.odi.org.uk/resources/download/135.pdf