Outcome classification: A framework to tally outcome statements across projects

It’s one thing to look at project and program ratings to see how well the stated objectives of a portfolio of projects are being met. But without an aggregate view of what these objectives are trying to achieve, it’s difficult to fully understand how well this portfolio is contributing to an organisation’s mission and wider development goals.

What’s missing is a better understanding of the links between project objectives and the types of outcomes that these are aiming to achieve.

Outcomes form part of everyday monitoring and evaluation practice, yet outcome statements are hard to aggregate and report on. While many development organisations report systematically on their reach, funding amounts, and performance data such as project ratings, they often report more anecdotally on their outcomes.

We decided to explore if there was a way to make sense, in an aggregated way, of the outcome statements contained in project documents, including completion reports with measures of outcome achievement. Was there a way that outcome statements could somehow be tallied, aggregated, compared within and across sectors?

This was important not only to enrich and complement the routine analysis of project performance ratings, but because we view outcomes as an essential part of what we, as evaluators, should be including in organizational performance reporting, and we needed a better way to do this.

The first challenge in this endeavour was to make sense of the different projects’ outcome statements that had previously been viewed as being too difficult to cluster and analyse. To assist with this, we developed the Outcome Classification Framework discussed below.

What is outcome classification?

Types of outcomes vary widely, both across and within different sectors. A program working within the health sector, for example, may have as an objective the delivery of an awareness raising program with the outcome of increased awareness of nutritional guidelines. Another program may have further reaching outcomes, such as reducing the stunting of children under five due to malnutrition.

The Outcome Classification Framework that we developed and applied categorises the outcomes of World Bank projects’ into four types, or ‘levels’, regardless of sector. Each of the four outcome levels correspond to a step in a theory of change for how the Bank Group’s work influences countries’ development outcomes, ranging from outputs at level 1 to early, intermediate, and long-term outcomes at levels 2, 3 and 4 (figure 1).

The four levels are largely distinct. Level 1 outputs include project deliverables, while level 2 outcomes are changes in the status quo that resulted from the level 1 deliverables. Hence, level 2 outcomes follow quite directly from project interventions and often focus on improved access, capacity, planning, and quality of public services — all of which represent relatively immediate benefits to beneficiaries. Level 3 outcomes follow indirectly from project interventions and are beyond the direct control of the project. At level 3, the level 2 outcomes have led to material improvements that solved development problems, causing sector wide ripple effects that benefit end beneficiaries. The ripple effects of level 4 outcomes are even wider. These are outcomes with systemic effects nationally or across sectors that contribute to general well-being. Level 4 outcomes correspond to the Sustainable Development Goals (SDGs) and similar wellbeing outcomes. This way, the levels indicate to what extent project outcomes have a line of sight to the SDGs and other higher-level outcomes.

We constructed the four categories by synthesising guidance on producing project results frameworks from the World Bank and other development agencies (see for example, USAID, SIDA and resultsbased.org (archived link)) and by testing the Outcome Classification Framework on actual World Bank Group projects. As such, the framework is largely a synthesis of existing practice.

Figure 1: Outcome classification framework

Outcomes Classification Framework,  Independent Evaluation Group (IEG)

What we found in applying the Outcomes Classification framework

Using the framework, we categorised the project objectives of all 989 World Bank projects during the fiscal years 2017 to 2020 that measured achievements and were validated by IEG. We also categorised other samples but have not discussed these here in order to simplify our explanations.

We found that project objectives cluster in clear outcome patterns depending on the sector and lending instrument. Most traditional projects (“investment projects”, not "budget support") target level 2 outcomes while most Development Policy Financing (“budget support”) operations target level 3 outcomes. It is established good practice to focus on outcomes, so it is positive that we found very few projects with level 1 objective statements.  

Figure 2: Outcome levels in a sample of World Bank projects

Compared to the outcomes found in Investment Financing Projects, there was a much larger percent of Development Policy Financing projects with outcomes at the higher outcome levels. Objective statements at outcome levels 3 and 4 were found in 54 and 19 percent of development policy projects, respectively. This is to be expected, as Development Policy Financing aims to induce change through policy, institutional, and governance reforms, with outcomes at these higher levels including macro fiscal stability, improved transparency and accountability, and increased domestic tax revenue.

Such outcomes are substantially different from most investment projects’ outcomes which, at level 3, include improved agricultural productivity, yields, and incomes; improved management of protected areas; climate resilience; and transport connectivity. A focus on sector wide change and end-beneficiaries characterises these types of level 3 outcomes.

In contrast to the outcomes of development policy projects, we found level 3 outcomes in only 26 percent of all investment project objectives and level 4 in 1 percent (figure 2). In contrast, level 2 outcomes were found in 72% of investment projects. These level 2 outcomes tended to focus on improving access to and quality of public services.

One possible explanation for the dominance of level 2 outcomes in investment projects is the way that ratings-based self-evaluation systems hold projects accountable for achieving objectives stated at approval (or formally revised subsequently). The need to demonstrate project achievements after closing may have led to a practice of stating objectives that can realistically be achieved and measured within the project’s lifetime. Projects can still have a line of sight to the SDGs and other higher-level outcomes in their motivation or other descriptive parts – they just tend to not have it in the formal objective statements.

We examined the relationship between projects’ stated outcome levels and end-of-project rating. Our analysis showed that while investment financing projects have markedly higher efficacy ratings than development policy financing projects, the outcome level of the project is not a statistically significant factor on the project rating. This finding of there being no systematic trade-off between projects’ outcome level and ratings runs counter to the assumption that one of the reasons for not setting higher level objectives is the risk of a lower rating (project ratings being of some relevance to World Bank internal incentives). The absence of a systematic trade-off between outcome level and ratings implies that many projects with higher-level objectives manage to achieve good outcome ratings, in part by having strong results frameworks to measure outcome achievement.

This raises the question of what is the “right” outcome level for projects? Objective setting needs to balance the opposing demands of realism and ambition. Realism demands that objectives be achievable, given the projects’ resources, timeline, and context. Objectives that are far removed from project interventions jeopardize the ability to show contribution and measure outcome achievement. Country and sector context, geographic scope, and beneficiaries are some of the context factors that also matter. On the other hand, ambition demands objectives with a line of sight to system-wide, transformative, or other important higher-level outcomes. Ambition also demands result frameworks that measure how the project changes beneficiaries’ conditions, with attention to gender and distributional aspects. Balancing realism and ambition requires judgment as well as dialogue between client counterparts and the World Bank.

Making use of the framework

The data gathered from applying the framework to World Bank project samples proved to be a useful complement to ratings-based performance metrics, leading to findings on how project objectives cluster in outcome patterns, among other findings. Comparability across sectors and countries is a key strength of the framework. The framework provides an entry point for comparative outcome analysis and the links between outcome levels and drivers of project ratings for different types of projects.

The framework is, however, a blunt tool. It focuses on stated and measured objectives, which may not be the same as the actual intended and unintended outcomes. It simplifies outcomes’ complex social realities into four categories that do not factor in context, so one country’s simple achievement could be an ambitious outcome in another. And whereas real-world outcomes are complicated processes of social change, the framework distils these down into four acontextual categories. We are well aware of the potential for misuse and misinterpretation. However, we were pleased that the findings, presented as part of the Results and Performance Report of the World Bank Group (2020), enabled greater focus on the organisations' outcomes and outcome orientation in discussions with counterparts and Executive Board members. We expect the framework could be useful to evaluators looking to enrich organisational performance assessments in contexts where such assessments cover many diverse projects.

We also hope the framework could help move forward discussions of what are the right levels of ambition to pursue in different contexts – an increasingly important question in light of global challenges that require system-wide and transformative action. The balancing act between realism and ambition is not simple, however we hope this framework will help organisations to look at the wider picture of their projects’ outcome-levels to better understand what their contributions are, and could be.

For more information, see the Results and Performance Report of the World Bank Group (2020).