RAPID Outcomes Assessment

Synonyms: 
ROA

RAPID Outcome Assessment (ROA) is a methodology to assess and map the contribution of a project’s actions on a particular change in policy or the policy environment. It is a flexible and visual tool that can be used in conjunction with other evaluation tools and options.

The option was developed by the RAPID (Research and Policy in Development) programme of the Overseas Development Institute (ODI) to analyse the impact of research on policy.

ROA draws significantly from the Outcome Mapping approach as it focuses on key actors that the project is directly influencing and the progressive changes in those actors. It also draws from other methodologies and approaches such as Episode Studies, which focuses on working backwards from a policy change to determine the factors that contributed to it; and Most Significant Change, which helps to identify and prioritise the key changes.

The ROA methodology has three main stages. The first stage is a preparation stage, during which a document review and a series of informal conversations are carried out to develop a draft picture of the project’s history and the intended changes. The second stage is the workshop during which the key policy change processes are identified by the stakeholders. The third stage involves a follow up process that allows the researchers to refine the stories of change, identifying key policy actors, events and their contribution to change.

Stage 1: Background research and preparation

1. The first step, as in any learning process, is attempting to develop a basic understanding of the situation. This will involve a review of project reports, project papers and research products, newspaper references/articles, relevant literature to the subject of the project and the policy environment before and after the project.

2. Conversations with relevant project staff and stakeholders will contribute to step one to identify overall policy objectives, the key actors and events that were targeted and the range of strategies used.

Stage 2: The ROA workshop

3. The aim of the workshop is to map behavioural changes in key actors and build a map of influences. The ROA team must consider who has to be invited to the workshops, how many workshops are necessary and how long will they last. The workshop will typically include the project team, key stakeholders and other external experts; the more diverse the participants the better.

4. The workshop will cover the following:

a. Defining the policy environment at the start and end of the project/period;

b. Identifying key actors, which may be individuals or institutions, and clustering into groups, for instance, ‘Civil society Organisations’, ‘Donors’, the ‘Private Sector’.

c. Characterising actors’ behaviour (i) now and (ii) at given start point in the policy process;

d. Establishing a timeline

e. Mapping key behaviour changes along the timeline;

f. Mapping (i) key project activities/changes, and (ii) external influences (events, influences, trends, shock) along the same timeline;

g. Determining the links/influences between the actors’ behaviour changes and the identified events, including project activities, external influences and other actors’ behaviour change.

Stage 3: Triangulate and refine conclusions

5. The ROA team should use the information gathered in the preliminary stages as well as the workshop to develop stories of change that describe the contributions of the project to the observed outcomes.

6. Use the timeline to identify informants to follow up with in-depth interviews. This will help to confirm the linkages and influences determined in the workshop and to assess the nature of the contributions to change.

Example

A joint study by the International Livestock Research Institute and the Overseas Development Institute applied the ROA option to investigate the influence on policy change of the Smallholder Dairy Project (SDP) in Kenya, a DFID funded research and development project which ran from 1997 - 2004.

The ROA was chosen because it allowed the researchers to approach the task from three angles:

  • A set of episode studies tracked back from identified changes in the dairy policy environment in Kenya during and after the implementation of the SDP, and analysed the catalysts and preconditions which led to the changes
  • A case studies tracked forward from the initiation of the SDP, highlighting key events in the project's history and identifying the immediate effects of its activities
  • An outcome mapping exercise analysed the changes in behaviour of the key actors involved: government departments and parastatials, research organisations, private sector and CSOs

The diagram below is the output from the stakeholder workshop and describes the outcomes observed and the contribution of various factors to these. The horizontal axis is time, with the far left column representing the time when the SDP project began and the far right column representing the time of the workshop. The very bottom row represents the project, with key milestones written on coloured squared. The row above that represents the external environment, with note-worthy events written on squares. Each of the other rows represents the different actors involved in the process, with key observed behaviour changes recorded on squares.

This data was collected using a variety of options: document analysis, key-informant interviews and field visits as well as focus group discussions at the workshop. The main aim of the workshop, however, was to draw on the collective experience and knowledge of the participants to analyse the data and develop causal inferences between the different events and the outcomes. These are illustrated by the coloured lines: blue indicates direct influence, red is indirect influence and green is external influence. Each of the lines is numbered to link it to a narrative description of the influence, the evidence for the connection and the discussion behind the placement of the line.

 

(Source: Leksmono et al (2006). Informal Traders Lock Horns with the Formal Milk Industry: The role of research in pro-poor dairy policy shift in Kenya. Overseas Development Institute http://www.odi.org.uk/resources/download/135.pdf)

Advice

Advice for CHOOSING this option (tips and traps)

Advice for USING this option (tips and traps)

  • This option requires involvement of key informants throughout the study. These should include representatives of key actors, project implementers and other experts with knowledge of the field.
  • The data from interviews and focus group discussions will almost certainly need to be triangulated, clarified and independently verified if it is to be used as evidence.

Resources

Guide

  • Making a difference: M&E of policy research.- The paper presents examples and approaches on conducting M&E of policy research from the current experience of a range of research institutes, think tanks and funding bodies.

Example

Sources

Hovland (2007). Making a difference: M&E of policy research. Overseas Development Institute. http://www.odi.org.uk/resources/download/281.pdf

Leksmono et al (2006). Informal Traders Lock Horns with the Formal Milk Industry: The role of research in pro-poor dairy policy shift in Kenya. Overseas Development Institute http://www.odi.org.uk/resources/download/135.pdf

 

Updated: 5th November 2014 - 3:42pm
A special thanks to this page's contributors
Author
Contributor
Research Fellow, Overseas Development Institute.
London.

Comments

Anonymous's picture
Monica

Hi,

I was wondering if you have examples of using the ROA for evaluating portfolios, i.e. multiple projects in the same thematic sector at the same time? If not, would the ROA still be an appropriate approach?

Thanks,

Monica

Patricia Rogers's picture
Patricia Rogers

Hi Monica,

Yes - check out this report on Impact evaluation for portfolio programmes on policy influence Reflections on the Indonesian Poverty Reduction Support Facility by Jessica Mackenzie and Simon Hearn  https://www.odi.org/sites/odi.org.uk/files/resource-documents/10463.pdf.

Here's an overview.

"Given that policy influence programmes need to be opportunistic and nimble, a rigorous measurement system to capture what works and why will never be perfectly comprehensive as it would make the programme too unwieldly or slow. Instead, successive programmes will need to have a light-touch system so they can remain flexible and agile. In addition to advocating for the use of light-touch systems, this paper recommends the consideration of six strategies (and guidance on how to apply them) to enhance planning, M&E of impact, discussed in Chapter 3. They are to:

1. develop appropriate logic models

2. collect observational data throughout implementation

3. develop stories of change or case studies

4. understand causal relationships without a counterfactual

5. purposefully select which activities to study

6. be explicit about how impacts will be valued across the portfolio."

Please share with us how you apply these to your work.

Kind regards,

Patricia Rogers

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.