Outcome harvesting

Outcome Harvesting collects (“harvests”) evidence of what has changed (“outcomes”) and, working backwards, determines whether and how an intervention has contributed to these changes.

Overview

Outcome Harvesting, as described in the 2012 guide by Ricardo Wilson-Grau and Heather Britt, is:

"…a method that enables evaluators, grant makers, and managers to identify, formulate, verify, and make sense of outcomes. The method was inspired by the definition of outcome as a change in the behavior, relationships, actions, activities, policies, or practices of an individual, group, community, organization, or institution.

Using Outcome Harvesting, the evaluator or harvester gleans information from reports, personal interviews, and other sources to document how a given program or initiative has contributed to outcomes. These outcomes can be positive or negative, intended or unintended, but the connection between the initiative and the outcomes should be verifiable." (p1)

An important feature of Outcome Harvesting is how outcomes are defined – observable and unobservable changes, both planned and emergent, in an individual, group, community, organisation, or institution resulting, at least in part, from an intervention of interest.

Reflecting Outcome Harvesting’s roots in Outcome Mapping, the definition of outcome contains an implicit causal chain from the project to the change agent the project seeks to influence (outside of the project) to other social actors that the project does not engage with directly. The following terms describe the roles of people involved in an outcome harvest:

• Change agent (informant): Individual or organisation that influences an outcome. The intervention that is being evaluated) works with them directly.
• Social actor: Individual, group, community, organisation, or institution that changes as a result of a change agent intervention.
• Verifier: The individual(s) who have knowledge about the outcomes and can be objective in reviewing the outcomes.
• Harvest user: The individual(s) who require the findings of an Outcome Harvest to make decisions or take action. This relatively small group plays a key decision-making role in the iterative design process of the harvest and is, therefore, distinct from larger groups of secondary users.
• Harvester: A person responsible for managing the Outcome Harvest, often an evaluator (external or internal).

In some cases, the harvester can engage directly with the social actors from whom they want to collect outcomes. When the harvester seeks to collect information about changes in social actors with whom the project does not engage directly, the harvester obtains information from project staff, the individual(s) or organisation(s) implementing the intervention, and change agents whose actions aim to influence the social actors.

Outcome descriptions are drafted to a level of detail considered useful for answering the harvesting questions. The descriptions may be as brief as a single sentence or as detailed as a page or more of text. At a minimum, descriptions should answer the following basic questions:

• Who changed?
• What was the observable change?
• When and where did the change take place?
• How did the project and change agents contribute to the change?

Questions may also include explanations of context, collaboration with or contributions from others to the specific outcome, diverse perspectives on the outcome, and the outcome's importance. The harvested information undergoes several rounds of revisions, including with social actors and those who provided the information, to make it specific and comprehensive enough to be verifiable. Information about select outcomes is validated by comparing it with information collected from other knowledgeable and authoritative but independent sources.

Finally, the validated outcome descriptions are analysed and interpreted – either as an individual outcome or as a group of outcomes – for their significance in achieving a mission, goal or strategy and are used to answer the harvesting questions.

Steps in the process

 

Outcome Harvesting consists of six steps detailed below. As an iteratively designed and participatory method, the steps are mutually informing and somewhat cyclical. The design outlined in Step 1 must be updated and refined at each subsequent step with input from the primary intended users and taking into account the information gleaned during the inquiry.

Each harvest should be customised to the specific needs of the primary intended users/uses and to reflect the project context. Application and adaptation of the six steps should be guided by the nine principles. The rigorous application of all six steps is necessary for a sound and credible outcome harvest.

1. Design the Outcome Harvest

The first step is to identify the primary intended users of the harvest and their principal intended uses for the harvest process and findings. Primary intended users, or a small group representing them, should be available to provide input during key points in the harvest process. Based on intended uses, the harvest users and harvesters agree on what needs to be known and write useful, actionable questions to guide the harvest (harvesting questions). For example, a useful question may be: What has been the collective effect of grantees on making the national governance regime more democratic, and what does it mean for the portfolio´s strategy? Then, they agree on what information is to be collected and from whom in order to answer the questions. At a minimum, this involves obtaining information about the changes in social actors and how the intervention influenced them. Practice principles 1, 2 and 4 are particularly useful here.

2. Review documentation and draft outcome descriptions

The key tasks include gathering data from documentation and project sources about outcomes and identifying

the key informants who will be contacted in Step 3. Harvesters consult with project staff as well as documentation to identify outcomes. From reports, previous evaluations, press releases and other documentation, harvesters identify potential outcomes (i.e., changes in individuals, groups, communities, organisations or institutions) and what the intervention did to contribute to them. For example, the change can be a president’s public commitment to being transparent (behaviour); two government agencies collaborating rather than competing (relationships); a minister firing a corrupt civil servant (action); the legislature passing a new anti-corruption law (policy); or a third successive government publishing its procurement records (practice). The influence of the change agent can range from inspiring and encouraging, facilitating and supporting, to persuading or pressuring the social actor to change.

The key elements of an outcome description are:

• Description of the change: Who changed? What specifically did the social actor do differently? When? Where?
• Contribution description: How did the project influence the outcome through influence on change agent(s)? Who did what with whom that plausibly influenced the outcome? What was done to influence the change (may be direct or indirect, intentional or not) Include when, where, who and what information.
• Significance statement: To whom is the outcome important and why?

Apply the four content principles when drafting outcomes:

1. Harvest social change outcomes: attitude, beliefs, or behaviour of a social actor influenced by the project through the actions of one or more change agents.
2. Formulate an outcome as a change: this change should be verifiable.
3. Establish plausible influence of the intervention: there must be a plausible contribution of the project to the outcome.
4. Ensure credible-enough outcomes: The harvest’s primary intended users determine how and when outcomes are deemed credible.

Outcome descriptions drafted in Step 2 are considered preliminary and will be confirmed or corrected in subsequent steps of the harvest.

3. Engage with social actors, change agents and key informants in formulating outcome descriptions

During step 2, the harvesters identify social actors, change agents and other key informants who have contributed to outcomes (both planned and emergent) and sample those to include in data collection. Harvesters engage directly with these key informants to collect data on whether and how they have influenced observable changes, including in other social actors. To avoid bias, harvesters ask open-ended questions about outcomes rather than limiting data collection to confirmation of outcomes documented in

Step 2 or included in project plans. At the end of this step, all outcome descriptions supported by evidence are finalised by applying the content principles.

4. Substantiate

Harvest users and harvesters review the final outcomes and select those to be verified to increase the accuracy and credibility of the findings. The harvesters share outcome descriptions with individuals who are independent of the intervention (third party) but knowledgeable about a specific outcome and the key informant’s/s’ contribution. Verification sampling strategies may vary widely. Apply content principle #9 when designing the verification sampling plan.

5. Analyse and interpret

Harvesters classify all outcomes, often in consultation with the primary intended users. The classifications are usually derived from the harvest questions; they may also be related to the objectives and strategies of either the implementer of the intervention or other stakeholders, such as donors. For large, multidimensional harvests, a database is required to store and analyse the outcome descriptions. Harvesters interpret the information and provide evidence-based answers to the harvesting questions. Analysis may involve mapping and documenting evidence of causal pathways from project activities and inputs to change agents and from change agents to observable changes in social actors (outcomes).

6. Support use of findings

Harvesters propose issues for discussion to harvest users grounded in the evidence-based answers to the harvesting questions. They facilitate discussions with users, which may include how they can use the findings.

Typically, feedback from step 4 (substantiation) and step 5 (analysis and interpretation) does not influence the earlier steps; feedback from step 6 (support of use) only affects step 5 (analysis and interpretation). Nonetheless, feedback from all the steps can, of course, influence decisions about future harvesting for either monitoring or evaluation purposes. The above steps are explained in more detail in Outcome Harvesting by Wilson-Grau R and Britt H (2012).

Principles for this approach

Five process principles and four content principles underpin outcome harvesting:

Process principles:

1. Facilitate usefulness throughout the harvest
2. Nurture appropriate participation
3. Coach human sources to formulate outcome statements
4. Strive for less because it will be more useful
5. Learn Outcome Harvesting experientially

Content principles:

1. Harvest social change outcomes
2. Formulate an outcome as an observable or otherwise verifiable change
3. Establish plausible influence of the intervention
4. Ensure credible-enough outcomes

Key characteristics/concepts of Outcome Harvesting

Outcome harvesting is characterised by:

• Iterative process: Key design decisions are made throughout the harvest as new information about outcomes emerges. For example, verification rationale and sampling may be developed after data collection. Previous steps can be revisited, and new questions formulated.
• Identifies early and otherwise hidden changes in the causal pathway: Outcome Harvesting is an important methodological breakthrough because it allows us to capture changes that are invisible to a project, because the social actors are either unknown or not accessible to the project. The project works directly with change agents who are outside and independent of the project. Many other methods can capture changes in those with whom the project works directly. Outcome Harvesting can capture changes farther down the causal pathway, even if we don’t know who social actors are in advance or how they have changed.
• Working backwards from observed outcomes: Outcome Harvesting asks ‘what has changed’ and works backwards to understand why the changes have occurred.

Causal Pathway features

How this approach might be used to incorporate features of a causal pathways perspective

A causal pathways perspective on evaluation focuses on understanding how, why, and under what conditions change happens or has happened. It is used to understand the interconnected chains of causal links which lead to a range of outcomes and impacts. These causal pathways are likely to involve multiple actors, contributing factors, events and actions, not only the activities associated with the program, project or policy being evaluated or its stated objectives.

Outcome Harvesting can be used in ways which incorporate the following features of a causal pathways perspective:

• Valuing actors’ narratives: Outcome Harvesting collects information about what has changed, why the change occurred (including the influence of the intervention being evaluated) and the importance of the change from social actors and/or change agents. Valuing actors’ narratives is demonstrated when harvest users who determine how and when outcomes are deemed credible and important include social actors and change agents.
• Addressing power and inclusion: The values and priorities of harvest users inform the iterative design of the harvest and harvesting questions. Power and inclusion can be addressed by involving people working to bring about change in considering whom to include as key informants and who is free to share stories on the record.
• Articulating explicit causal pathways: Outcome statements describe how project contributions influence change agents and how change agents then influence changes in social actors, regardless of whether these causal pathways were included in the theory of change. In this sense, Outcome Harvesting can contribute to building or revising a theory of change.
• Paying attention to a range of outcomes and impacts: Outcome Harvesting is specifically designed to capture outcomes that cannot be predicted in advance and are not visible to implementers. Social actors and change agents working directly with implementers report on observable changes in social actors not known to or accessible to implementers.
• Understanding contextual variation: Outcome statements may include information about context.
• Using an iterative, bricolage approach to evaluation design:The steps should be implemented iteratively; data collected in one step can generate new questions and further data collection.
• Drawing on a range of causal inference strategies: Relies on verifiable statements about how change agents have influenced social actors to achieve outcomes.
• Taking a complexity-appropriate approach to evaluation quality and rigour: Findings are substantiated by independent informants with knowledge of outcomes and the contribution of change agents. Multiple data sources are used to find evidence of outcomes. Utilisation facilitated by involving users in defining purpose.

Background

History of this approach

The approach is inspired by Outcome Mapping and informed by Utilisation-Focused Evaluation. It was developed by Ricardo Wilson-Grau and colleagues* who, as co-evaluators, applied it in different circumstances and further refined it over many years of evaluation practice. An active community of Outcome Harvesting practitioners shares and generates ongoing learning.

*Barbara Klugman, Claudia Fontes, David Wilson-Sánchez, Fe Briones Garcia, Gabriela Sánchez, Goele Scheers, Heather Britt, Jennifer Vincent, Julie Lafreniere, Juliette Majot, Marcie Mersky, Martha Nuñez, Mary Jane Real, Natalia Ortiz, and Wolfgang Richert.

Examples

Outcome Harvesting was used to evaluate the impacts of OIKKO (Unity in Bangla) – a three-year project implemented between March 2015 and February 2018 with the overall objective of a strong and united civil society that promotes the implementation of fundamental labour rights in the ready-made garment (RMG) sector in Bangladesh. The project involved forming solidarity groups of women RMG workers, larger community worker associations, community support groups, a national platform, training, publications and a national convention. The evaluation was internal with the support of an external evaluator experienced in outcome harvesting.

The evaluation sought to understand how the program contributed to outcomes of interest, including increased ability, confidence and motivation of women workers to raise demands in the workplace or community, whether individually or collectively, and the improved responses of employers to RMG work better understand the most important pathways of change, and to identify negative or unexpected outcomes.

Fifteen findings were reported from 46 outcomes harvested during the evaluation.

The evaluation found that the scope of actions taken by workers and access to rights and entitlements gained was far beyond what CARE has previously contributed to through worker training and group-based approaches, suggesting that CARE should invest in EKATA as an effective model for promoting access to workers’ rights. it also identified some important negative outcomes – in particular, the fact that encouraging workers to raise demands put them at considerable risk.

The evaluation report, which provides instructive detail about the evaluation design and process, including reflections on the approach is available on the outcome harvesting website.

Outcome Harvesting has proven to be a useful approach in a variety of locales and for diverse interventions. Some examples are:

• Summative evaluation of the Oxfam Novib’s Global Programme 2005-2008, a €22 million investment to support 38 grantees working on sustainable livelihoods and social and political participation. Documentation of more than 300 outcomes from 111 countries.
• Retrospective 'Outcome Harvesting': Generating robust insights. Report on the evaluation experience of the BioNET global network to promote taxonomy for development. Identification and documentation of 200 emergent outcomes of BioNET.
• World Bank Institute Cases in Outcome Harvesting. Ten pilot experiences identify new learning from multi-stakeholder projects to improve results. There is an average of 30 outcomes per pilot.

Advice for choosing this approach

What types of projects and programs would Outcome Harvesting be appropriate for?

Outcome Harvesting is particularly suitable to assess social change interventions, innovation and development work, or other interventions in complex systems when it is not possible to define concretely most of what an intervention aims to achieve or even what specific actions will be taken over a multi-year period.

Outcome Harvesting is well-suited to projects where emerging outcomes and unpredictable causal pathways are likely to comprise a sizeable proportion of the project or have a significant influence.

Outcome Harvesting is not intended for projects or project components that are well-supported by evidence, or that closely follow a theory of change.

Outcome Harvesting is appropriate for identifying and verifying outcomes, including unintended outcomes. It is a useful, commonsense approach that easily engages informants and is designed to generate concrete evidence to inform decisions about future actions.

In complex environments, objectives and the paths to achieve them are largely unpredictable and predefined objectives and theories of change must be modified over time to respond to changes in the context. Outcome Harvesting is particularly appropriate in these more dynamic and uncertain environments in which unintended outcomes dominate, including negative ones.

What types of evaluations is Outcome Harvesting appropriate for?

Outcome Harvesting can be used for different purposes - for ongoing developmental, mid-term formative, and end-of-term summative evaluations. It can be used by itself or in combination with other approaches. Outcome Harvesting tracks the changes in the attitudes, beliefs and behaviours of social actors influenced by an intervention; however, it is designed to go beyond this and support learning about those achievements.

Outcome Harvesting is designed for situations where decision-makers (as “harvest users”) are most interested in learning about what was achieved and how. In other words, there is an emphasis on effectiveness rather than efficiency or performance. The approach is also a good fit when the aim is to understand the process of change and how project contributions, social actors and change agents contribute to outcomes, and how individual outcomes contribute to broader changes.

The timing of “the harvest” depends on how essential the harvest findings are to ensure the intervention is heading in the right direction. If the certainty is relatively great that doing A will result in B, the harvest can be timed to coincide with when the results are expected. Conversely, if much uncertainty exists about the results that the intervention will achieve, the harvest should be scheduled as soon as possible to determine the outcomes that are actually being achieved and the progress being made.

What level and type of resources are required for Outcome Harvesting?

Skill and time, as well as timeliness, are required to identify and formulate high-quality outcome descriptions. Depending on the period covered and the number of outcomes involved, the approach can require a substantial time commitment from informants. Harvests can be scaled to available time and resources. Outcome Harvesting principle #4 reminds harvesters to emphasise learning over collecting a large number of outcomes: “strive for less because it will more useful”. To reduce the burden of time on informants, outcomes may be harvested in an ongoing way (monitoring) or periodically (monthly, quarterly, biannually, or annually), and findings may be substantiated, analysed or interpreted less frequently.

How might Outcome Harvesting be part of an effective overall evaluation design?

Outcome Harvesting can complement theory-based approaches that focus on the extent of planned changes by identifying unintended changes and the importance of these changes to harvest users. Outcome Harvesting supports learning and reflection to inform adaptive management.

Advice for using this approach effectively

A highly participatory process is a necessity for a successful Outcome Harvesting process and product.

“Harvest users” are individuals or organisations requiring the findings to make decisions or take action. They should be engaged throughout the process. These users must be involved in making decisions about the design and re-design of the approach as the process and the outcomes come to light. They also determine how and when outcomes are deemed credible. Also, the principal uses for the harvest may shift as findings are generated which, in turn, may require re-design decisions.

Either an external or internal person (“the harvester”) is designated to lead the Outcome Harvesting process. Harvesters facilitate and support the appropriate participation of the harvest users and ensure that the data are credible, the criteria and standards to analyse the evidence are rigorous, and the methods of synthesis and interpretation are solid. Facilitating a participatory and iterative design process requires skill and experience. Harvesters also require data collection and analysis skills. Various data collection methods such as interviews and surveys (face-to-face, by telephone, by e-mail), workshops and document review can be used.

Participation of the social actors and/or change agents who influenced the outcomes is crucial. The harvester needs to engage informants who are knowledgeable about what the intervention has achieved and how and who are willing to share, for the record, what they know. Field staff who are positioned “closest to the action” are often best positioned to identify social actors and change agents to serve as respondents. Project staff may also provide useful information about outcomes, project contribution, and significance.

To reduce the risk of positive bias, it is important to also capture information about negative outcomes or expected outcomes that didn’t occur. No change or maintenance of the status quo can be significant outcomes.

Starting with the outcomes and working backwards represents a new way of thinking about change for some participants. Consequently, the rationale for the process needs to be clearly explained.