Indicator framework for OSF’s Economic Justice Program

This resource is the framework that the MEL team in the Open Society Foundations’ Economic Justice Program developed to compile, categorize, and make sense of a range of different indicators across the program’s grant-making portfolio.

This resource is part of the MEL toolkit for grantmakers and grantees resource collection.

Authors and their affiliation

Andrea Azevedo and Jay Locke (Economic Justice Program MEL Team, Open Society Foundations), Laura Rana and Emma Broadbent (Consultants)

Key features of this indicator framework

Understanding the collective impact of a portfolio of diverse grants can be challenging for donors. To make it easier to aggregate and analyze this data from different sources and on different activities and outcomes, donor organizations often require grantees to use standardized indicators. But imposing indicators is fraught with problems, creating an effort-reward mismatch and deepening existing imbalances of power between donors and grantees. This is a major obstacle to good MEL on all sides with pervasive effects, steering measurement priorities and approaches towards donor interests instead of the needs of the field.

This resource is the framework that the MEL team in the Open Society Foundations’ Economic Justice Program developed to compile, categorize, and make sense of a range of different indicators across the program’s grant-making portfolio.

In creating this indicator framework, the team looked first at what the portfolio leads wanted to know—that is, the questions they wanted to answer about the work. Having translated these questions into indicators, the MEL team identified a small number that would require information not likely to be available from grantee reporting; this information would need to be gathered from existing secondary data, such as national statistics, or further research. Data for most indicators, however, was expected to come from grantees.

The MEL team was committed to giving grantees flexibility in what to measure while also recognizing the limitations of relying on a wide variety of grantee indicators and the burden of work it would create internally. To overcome this, the team created outcome “baskets” to which individual grantee indicators (set by the grantees) would then be mapped. By clustering grantee indicators, these portfolio-level “baskets” would gather relevant data that the program and MEL staff could analyze periodically across the portfolio and beyond the individual grant.

During this process, the MEL team noticed that, in some cases, grantees were asking similar field-wide questions and were producing relevant, high-quality data to answer them (for example, using national data to elaborate reports on a particular theme, such as gender budgeting). In these cases, grantee indicators were directly incorporated into the portfolio framework as representative of changes in the field.

The indicator framework is a curated database of indicators that categorizes indicators by grant-making portfolio, goal, milestone, outcomes area, data source, and a range of additional characteristics that could be used to analyze and cluster activities and results that spanned grantees but sought to influence similar changes.

How have you used or intend on using this indicator framework?

The EJP MEL team used this indicator framework to identify how different EJP grantees were thinking about tracking and measuring progress and where commonalities across these measures could help both the EJP team and partner organizations understand how different organizations’ work might be aligned and moving toward the same goals and milestones. The database also served as a reference for organizations seeking to identify indicators for their work; while not all organizations would be able to find a “ready-to-use” indicator in this database, the contents provided ideas for those struggling to decide how to measure their work.

Why would you recommend it to other people?

Donors and partners working in the economic justice field will find this resource valuable in presenting ideas for indicators to measure progress within different areas of economic justice. For those working outside of this field, the categories of outcome areas (such as narrative change, policy change, and changes in practice) may provide ideas for how to articulate and position indicators or structure an indicator framework.

Sources

Azevedo, A., Locke, J., Rana, L. & Broadbent, E. (2020). Indicator framework for OSF’s Economic Justice Program. Retrieved from https://www.betterevaluation.org/tools-resources/indicator-framework-for-osfs-economic-justice-program