Missing link: Sustained and emerging impact evaluation

This blog post by Jindra Cekan and Laurie Zivetz of Valuing Voices discusses the need for post-project impact evaluations. An area that needs more attention, BetterEvaluation will be working with Valuing Voices over the next couple of months to expand the available information and resources for this type of evaluation.

The question of what is impact is one that has been coming up again and again recently. Simon Hearn's recent blog on BetterEvaluation (and his accompanying Methods Lab paper) began unravelling the many ways we use the term. Simon notes that in addition to the range of definitions of impact, our industry has a growing appetite to demonstrate the impact of our international development projects with evidence. In a blog post last year, Developing a Research Agenda for Impact Evaluation, Patricia Rogers and greet Peersman note that current impact evaluations tend to look “only at relatively short-term, intended direct effects” and attention to unintended or unexpected (emerging) impacts of our projects remains undeveloped. While organisations such as 3ie and J-Pal have risen to prominence for their work on impact evaluation, the different views surrounding what constitutes a sustained impact in these organisations are important (read the Valuing Voices blog post on the differences between our definition and theirs, and the implications of this).

The preoccupation with short-term impacts represents a serious gap in evaluation practice, theory and design. Returning 2-10 years post-project offers an opportunity to assess whether improvements- such as those in organisational efficiencies, community infrastructure, knowledge, behaviour change, livelihoods- that may have been planned and shown at the end of the project cycle, actually endured.  It also provides a chance to understand whether other unintended impacts emerged over time as a result of a project or program or participants’ efforts in the intervening years after the project ended. How can we claim we are doing “sustainable development” if we do not return to assess sustainability that we either envisioned or that emerged from people’s own efforts?

This gap presents serious challenges to our ability to be genuinely accountable or to learn and improve - also fundamental to effectively meeting the Sustainable Development Goals. We believe that funding, doing and learning from Sustained and Emerging Impact Evaluations (SEIE) needs to become a standard part of the program cycle. This is key to doing sustainable development.  Building an evidence base and generating lessons across projects and sectors, and from design to implementation, monitoring to evaluation promise benefits in terms of better design and implementation of projects, more robust, accountable partnerships, and a path to foster true country ownership.

The numbers are staggering. The OECD tells us that over $5 trillion in foreign aid has been given since 1970 (excluding private donations). Over $1.5 trillion of that has been given by the EU and USAID since 2000 alone and even last year $137 billion was spent worldwide. At Valuing Voices, we have reviewed hundreds of documents tagged as ‘post-project’ and ‘ex-post’ evaluation from nearly 30 international development and implementer organisations. Most of these documents only suggest that such evaluations be done; a few are desk studies. Only a handful (some 1% of all projects actually evaluated) have actually returned to the field post-project, and systematically asked participants and partners what they had sustained themselves 2-10 years after the project ended.  Even fewer of these have delved into why some outcomes were sustained, and asked questions about what other impacts or innovations emerged.  It seems that much of the ‘sustainability’ focus is on the continuation of funding for projects, rather than assessing impact, and including the efforts and hopes of our stakeholders.  Curiosity about what might be done better seems to stop at the end of the resourced project cycle.

Valuing Voices Full-Cycle Development

The 19 organisations that we have found that have done one or more such evaluations deserve to be strongly celebrated as catalytic learning organisations. They are at the vanguard of redefining ‘success’ through our participants’ and partners‘ eyes and improving our programming accordingly.  We have piloted this approach in a number of post-project sustained impact evaluations that found important lessons for design with follow-on funding as well as for monitoring and implementation.  We have documented the tip of the sustained impact iceberg, including an excellent study by USAID that highlighted four essential elements for successful exit (resources, capacity, linkages and motivation). We discuss some of the assumptions many of us make about barriers to doing more such evaluations, Including needed improvements to our M&E systems, how to forge stronger partnerships and country ownership.

We ask you to join Valuing Voices and BetterEvaluation in advocating creating new success measure in international development: sustained and emerging impact evaluations.  We ask you to fund and do these SEIEs and share them for inclusion in our database. Such learning should not only be standard in our industry, but the lessons from them to be a globally available resource. We envision creating regional centres of analysis of learning and country-ownership, with data available long after any project ends. We ask you to help us put participant feedback at the centre of all of our work so we can better learn about sustained and emerging impact, foster design for success, implement for national ownership of projects long after we are gone. That is why we do what we do, isn’t it?

Have you been involved in a post-project evaluation of sustained and emerging impacts? Or do you know of any good examples or have some advice on conducting one?