RAPID outcome assessment (ROA) is a method to assess and map the contribution of a project’s actions on a particular change in policy or the policy environment.
It is a flexible and visual tool that can be used in conjunction with other evaluation tools and methods.
The method was developed by the RAPID (Research and Policy in Development) programme of the Overseas Development Institute (ODI) to analyse the impact of research on policy.
ROA draws significantly from the Outcome Mapping approach as it focuses on key actors that the project is directly influencing and the progressive changes in those actors. It also draws from other methodologies and approaches such as Episode Studies, which focuses on working backwards from a policy change to determine the factors that contributed to it; and Most Significant Change, which helps to identify and prioritise the key changes.
The ROA methodology has three main stages. The first stage is the preparation stage, during which a document review and a series of informal conversations are carried out to develop a draft picture of the project’s history and the intended changes. The second stage is the workshop during which the key policy change processes are identified by the stakeholders. The third stage involves a follow-up process that allows the researchers to refine the stories of change, identifying key policy actors, events and their contribution to change.
Stage 1: Background research and preparation
1. The first step, as in any learning process, is attempting to develop a basic understanding of the situation. This will involve a review of project reports, project papers and research products, newspaper references/articles, relevant literature to the subject of the project and the policy environment before and after the project.
2. Conversations with relevant project staff and stakeholders will contribute to step one to identify overall policy objectives, the key actors and events that were targeted and the range of strategies used.
Stage 2: The ROA workshop
3. The aim of the workshop is to map behavioural changes in key actors and build a map of influences. The ROA team must consider who has to be invited to the workshops, how many workshops are necessary and how long will they last. The workshop will typically include the project team, key stakeholders and other external experts; the more diverse the participants the better.
4. The workshop will cover the following:
a. Defining the policy environment at the start and end of the project/period;
b. Identifying key actors, which may be individuals or institutions, and clustering into groups, for instance, ‘civil society organisations’, ‘donors’, the ‘private sector’.
c. Characterising actors’ behaviour (i) now and (ii) at a given start point in the policy process;
d. Establishing a timeline
e. Mapping key behaviour changes along the timeline;
f. Mapping (i) key project activities/changes, and (ii) external influences (events, influences, trends, shock) along the same timeline;
g. Determining the links/influences between the actors’ behaviour changes and the identified events, including project activities, external influences and other actors’ behaviour change.
Stage 3: Triangulate and refine conclusions
5. The ROA team should use the information gathered in the preliminary stages as well as the workshop to develop stories of change that describe the contributions of the project to the observed outcomes.6. Use the timeline to identify informants to follow up with in-depth interviews. This will help to confirm the linkages and influences determined in the workshop and to assess the nature of the contributions to change.
A joint study by the International Livestock Research Institute and the Overseas Development Institute applied the ROA method to investigate the influence on policy change of the Smallholder Dairy Project (SDP) in Kenya, a DFID funded research and development project which ran from 1997 - 2004.
The ROA was chosen because it allowed the researchers to approach the task from three angles:
- A set of episode studies tracked back from identified changes in the dairy policy environment in Kenya during and after the implementation of the SDP, and analysed the catalysts and preconditions which led to the changes
- A case study tracked forward from the initiation of the SDP, highlighting key events in the project's history and identifying the immediate effects of its activities
- An outcome mapping exercise analysed the changes in the behaviour of the key actors involved: government departments and parastatals, research organisations, private sector and CSOs
The diagram below is the output from the stakeholder workshop and describes the outcomes observed and the contribution of various factors to these. The horizontal axis is time, with the far left column representing the time when the SDP project began and the far right column representing the time of the workshop. The very bottom row represents the project, with key milestones written on coloured squared. The row above that represents the external environment, with note-worthy events written on squares. Each of the other rows represents the different actors involved in the process, with key observed behaviour changes recorded on squares.
This data was collected using a variety of methods: document analysis, key-informant interviews and field visits as well as focus group discussions at the workshop. The main aim of the workshop, however, was to draw on the collective experience and knowledge of the participants to analyse the data and develop causal inferences between the different events and the outcomes. These are illustrated by the coloured lines: blue indicates direct influence, red is indirect influence and green is external influence. Each of the lines is numbered to link it to a narrative description of the influence, the evidence for the connection and the discussion behind the placement of the line.
Source: Leksmono et al. (2006).
Advice for choosing this method
- This method helps address the attribution problem by mapping the effect of multiple factors and actors on observed or measured outcomes.
- This method requires experience with data collection (e.g. interviews, surveys, focus groups), analysis (e.g. episode studies, case studies) and facilitation skills.
Advice for using this method
- This option requires involvement of key informants throughout the study. These should include representatives of key actors, project implementers and other experts with knowledge of the field.
- The data from interviews and focus group discussions will almost certainly need to be triangulated, clarified and independently verified if it is to be used as evidence.
Hovland (2007). Making a difference: M&E of policy research. Overseas Development Institute. https://odi.org/en/publications/making-a-difference-me-of-policy-research/
Leksmono et al (2006). Informal Traders Lock Horns with the Formal Milk Industry: The role of research in pro-poor dairy policy shift in Kenya. Overseas Development Institute https://odi.org/en/publications/informal-traders-lock-horns...
'RAPID outcomes assessment' is referenced in:
- Rainbow Framework :
- Rainbow Framework :