Demonstrating outcomes and impact across different scales

In this guest blog, Jo Hall discusses how evidence of outcomes and impact can be better captured, integrated and reported on across different scales of work.

It is common practice for organisations to view the impact of their activities on a project by project basis. But to gain deeper insights into the collective contribution of multiple projects, or to learn more about what is working well or not across different projects, such as in the same geographic area or sector, requires thinking, analysing and evaluating at different levels. For example, the Norwegian Government assessed the overall and wider impact of NGOs and CSOs on development and their long-term contribution to poverty reduction in three countries and found some higher-order results that the individual organisations were unaware of. In another example, a group of Australian NGOs synthesised findings of multiple education projects and programs across multiple countries and agencies to demonstrate the unique and necessary contribution of NGOs to education. How to go about evaluating across different scales beyond single projects is relatively unchartered territory and not without its challenges.

The Australian Council for International Development (ACFID) and the Research for Development Impact (RDI) Network commissioned a research report Demonstrating outcomes and impact across different scales to show how evidence of outcomes and impact can be better captured, integrated and reported on across different scales of work for Australian NGOs working in international development. The report looks at some of the different methods available to reporting at different scales ‘beyond the project’. This is a complicated endeavour because monitoring and evaluation systems, which have developed rapidly over the last twenty years, are not yet delivering a clear picture on what is working well and what is not working well in different contexts in international aid even at a project level.

Drawing on five case studies, the report considers the strengths and limitations of three broad and overlapping groups of approaches and methods, taking account of seven factors that are important for NGOs to consider when capturing and reporting on outcomes across different scales:

• Being clear on the purpose and questions and making use of the information
• Applying fit-for-purpose methods in fit-for-purpose ways
• Adopting methods and approaches that address complexity
• Considering the needs of all partners, including locally
• Capturing the distinctive contribution of the NGOs
• Not being overly complicated, technocratic or exacerbating fragmentation
• Having adequate resourcing to meet the purpose

The research is not a comprehensive treatise on the subject, rather it focuses on describing and analysing some of the major approaches and methods that are likely to be relevant for Australian NGOs and draws on numerous examples, including the five case studies, to illustrate.

Selecting and implementing the most appropriate methods or combinations of methods is both a technical and political issue. Many NGOs have a tendency is to think of their own agencies, reputations and public relations above the need for local engagement, collaboration and collective thinking about the contributions of NGOs and partners to achieving development impact. In part, the NGOs are operating in a competitive funding environment which is not conducive to collaboration. In part, many operate from a project mindset that detracts from thinking at more strategic levels.

Quantifying outcomes and impact in both quantitative and qualitative terms is very important. But the popularity of an agency-centric use of aggregating indicators among international NGOs is a cause for some unease because on their own they do not address complexity, are often not considering the needs of local partners and are not helping to further a global understanding of the distinctive contribution of NGOs. One of the issues is the isolated use of indicators in reporting.

Approaches used for integrating and reporting across different scales of work also need to address complexity or they risk perpetuating over-simplified understandings of program logic and development and will not be useful. ‘Thinking and working politically’ suggests fewer pre-defined results, mixed approaches to monitoring and evaluation and the need for enhanced collaboration.

An important consideration in capturing the distinctive contribution of NGOs is a focus on the nature of the intervention and how this has contributed to outcomes and impact. This requires a mix of methods and approaches that not only focus on the outcomes and impact but also include some explanatory power. Many of the research and evaluative approaches described, such as realist evaluation and realist synthesis, seek to understand the mechanisms of how the particular change has come about.

ACFID and the RDI Network are driving follow-up and identifying emergent opportunities for the Australian NGOs to take up, starting with a national workshop on 28 August 2018. See the ACFID website for further news.

Read more