52 weeks of BetterEvaluation: Week 16: Identifying and documenting emergent outcomes of a global network
Global voluntary networks are complex beasts with dynamic and unpredictable actions and interactions. How can we evaluate the results of a network like this? Whose results are we even talking about? This was the challenge facing BioNET when they came to the end of their five year programme and is the subject of the second paper in the BetterEvaluation writeshop series, which we want to introduce in this weeks' blog.
Three programme members, Kornelia Rassmann, Richard Smith, John Mauremootoo and the lead-evaluator, Ricardo Wilson-Grau, describe their use of Outcome Harvesting to evaluate BioNET-International, a global, voluntary network for capacity building in natural science, conservation and agriculture. They discuss the evaluation and the methods they use in the short video below and the following blog post.
Click here to open this video in YouTube. Duration: 9 minutes.
Outcome Harvesting is an approach developed by Ricardo Wilson-Grau and colleagues over the last 10 years which uses the principles of Outcome Mapping. As the name suggests, it “harvests” outcomes (as defined in Outcome Mapping) – but in retrospect. That means it collects evidence of what a programme or project has achieved in a given period of time, then orders and substantiates the evidence and finally evaluates how far the intervention contributed to change. It is not essential that the development programme under evaluation has been monitored for outcomes as they emerged (but as you’ll see below, it helps).
The BioNET Global Programme 2007 to 2011 had been monitored using a logframe-based approach; however, in a network as complex as BioNET – made of diverse, voluntary memberships with dynamic and unpredictable actions and interactions – we found limitations to this approach. Knowing about the advantages of Outcome Mapping particularly for international networks, the staff of the BioNET Global Secretariat introduced this approach in 2010 for planning the network’s next phase. We immediately resonated with the Outcome Mapping approach and recommended to use Outcome Harvesting also for BioNET’s summative evaluation, which took place in late 2010 and early 2011, at the end of BioNET’s Global Programme.
The Outcome Harvesting approach proved to be a successful way to deal with the complex planning, monitoring and evaluation situation found in the BioNET network. The process harvested almost 200 outcomes which provided a rich resource of succinct “achievement statements”, not only for use in the evaluation, but also as potential PR material for external uses. This included unforeseen outcomes which a conventional evaluation, focusing on what was achieved against what was planned, would not have picked up on.
The evaluation gained further weight and credibility through a “substantiation” process, where several independent external sources went on public record, stating how far they agreed or disagreed with the outcomes harvested. Substantiation thus contributed to the usefulness of the evaluation: it served as a promotional element for the network, strengthening linkages to internal and external partners who were invited to participate in the evaluation process. Another very positive effect of the intensely participatory Outcome Harvesting approach was that it strongly engaged several different groups of stakeholders and encouraged learning at many different levels, promoting a high level of self-reflection and helping to familiarise network actors with the concept of 'contribution' rather than 'attribution' that is at the heart of the tool.
What has to be considered when using the Outcome Harvesting approach is, however, that participation draws heavily on the time and resources of internal staff. This starts with several rounds of drafting and revising the evaluation design together with the evaluators and does not end with the laborious “harvesting” of evaluation data – the drafting and reviewing of outcome, contribution and significance statements. Particularly for this part of the process it is very helpful if the internal staff engaged in the evaluation are familiar with the concept of “contribution” and how to draft outcome statements accordingly (i.e. “who precisely did what, when and where”).
The BioNET evaluation showed that even when the users of the evaluation lack extensive knowledge of Outcome Mapping or Outcome Harvesting, the approach can be very successful. However, it would facilitate the process and save time if the method was better understood and ideally if the monitoring and evaluation system already in place was Outcome Mapping-based. An Outcome Harvest requires a high level of engagement of internal staff as well as essential engagement with external substantiators. As a result, an Outcome Harvesting generates concrete, substantiated findings on the outcomes you have achieved and how you achieved them. These findings will have ownership throughout your organisation and your external stakeholders.
In the BioNET case the evaluation created significant insights which helped to shape BioNET’s future, as well as a rich resource of succinct ‘achievement statements’ to use for promotional purposes.
More on the BioNET evaluation
Outcome Mapping newsletter article: http://www.outcomemapping.ca/resource/resource.php?id=356
BioNET evaluation report: http://www.outcomemapping.ca/resource/resource.php?id=349
Other Outcome Harvesting evaluations
Written by Kornelia Rassmann with Richard Smith, John Mauremootoo and Ricardo Wilson-Grau.
Image credit: Neil Palmer (CIAT)