Footprint evaluation

Footprint evaluation focuses on the ‘footprint’ that human systems make on natural systems. This requires attention to the nexus between human systems and natural systems. Footprint evaluation is grounded in the premise that all evaluations should include consideration of environmental sustainability, even when this is not a stated goal of the intervention. This is so that decision-making can take into account the potential and actual impacts of planned interventions (projects, programs, policies) on the environment. 

Why footprint evaluation is needed

The world is faced with numerous environmental crises with the potential for global catastrophe, including climate change, pollution, loss of biodiversity, and deforestation.  In order to address, mitigate and avoid these, it is essential that decision-making takes into account the potential and actual environmental impacts of planned interventions, and that evaluations of interventions provide this information.  However, although the OECD-DAC evaluation criteria include "significant environmental impacts" in the scope of impact evaluation,  most evaluation usually fails to do so, and specialist environmental evaluation often focuses on narrow technical questions rather than using evaluation concepts, processes and methods to answer evaluative questions in ways that can support improved decision-making.

Stocktakes of the extent to which environmental considerations were included in evaluations were conducted by the UN Evaluation Group and the Canadian Evaluation Society in 2020. The UN Evaluation Group found that there was very limited guidance on environmental considerations and that environmental considerations were less likely to be considered in evaluations. The Canadian Evaluation Society found that it was rare for the natural system to be considered in federal and other evaluations and that there was negligible North American intellectual infrastructure on natural systems, environment and sustainability. 

Improving evaluations to be able to include ‘footprint evaluation’ will require a combination of supply (more people who can do this), demand (more people asking for this in evaluations) and enabling environments (setting up systems to provide incentives, opportunities and capacities to do this). 

What might it take?

Some of the challenges identified so far by the footprint evaluation project include:

  • How do we get environmental considerations on the agenda for an evaluation?

  • How do we include environmental considerations without breaking the evaluation budget?

  • How do we effectively work across disciplinary boundaries?

  • How do we include environmental considerations in a meaningful way ( while avoiding becoming another cross-cutting issue tick the box exercise)?

  • What is involved in being transparent about decisions regarding trade-offs between costs and benefits for natural and human systems?

  • What can we learn from other efforts to include particular considerations in all evaluations (especially gender, human rights, and fragility/conflict)?

These and other challenges, as well as opportunities, are being explored in the footprint project. In this update options for getting environmental considerations on the agenda are discussed.

How do we get environmental considerations on the agenda for an evaluation?

One of the challenges identified by the footprint project so far is how to get the consideration of natural systems on the agenda for an evaluation. This is an issue for both evaluation commissioners and evaluators raising questions about what can be done by those developing Terms of Reference if environmental sustainability is not one of the stated objectives of a program, project or policy? And, what can be done by evaluators if environmental sustainability is not explicitly included in the Terms of Reference for an evaluation?

Possible options include:

Drawing on the OECD-DAC Criteria

The OECD-DAC criteria were revised in 2019 aim to support better evaluation, leading to better policies to advance both the 2030 Agenda for Sustainable Development (including the Sustainable Development Goals) and the Paris Agreement within the United Nations Framework Convention on Climate Change. The criteria are: relevance, effectiveness, efficiency, impact, sustainability and coherence.

The criteria that are likely to be particularly useful for justifying the consideration of natural systems are:

Relevance  - defined by the OECD-DAC as: “The extent to which the intervention objectives and design respond to beneficiaries’, global, country, and partner/institution needs, policies, and priorities, and continue to do so if circumstances change.” Where respond to is defined by the OECD-DAC as: “that the objectives and design of the intervention are sensitive to the economic, environmental, equity, social, political economy, and capacity conditions in which it takes place.… Relevance assessment involves looking at differences and trade-offs between different priorities or needs.”

Evaluations that consider both natural and human systems will often need to consider how apparently competing needs and priorities are assessed and managed.

Coherence is defined as: “The compatibility of the intervention with other interventions in a country, sector or institution.” Coherence refers to “the extent to which other interventions (particularly policies) support or undermine the intervention, and vice versa. It includes both internal and external coherence.”

The explanatory note clarifies that the coherence criterion “…encourages analysis of the consistency of the intervention with the actor’s own commitments under international law or agreements” and points out that “International norms and standards may also be assessed under relevance from the viewpoint of responsiveness to global priorities, which is a complementary angle.”

Impact is defined as “The extent to which the intervention has generated or is expected to generate significant positive or negative, intended or unintended, higher-level effects.” In particular the “…the holistic and enduring changes in systems or norms, and potential effects on people’s well-being, human rights, gender equality, and the environment.”

Sustainability is defined asThe extent to which the net benefits of the intervention continue, or are likely to continue.” The explanatory note clarifies that sustainability has various dimensions (financial, economic, social and environmental).

A natural systems lens on the sustainability of impacts involves considering resiliency and any risks to the intervention’s impacts that are likely to accompany climate change and other sources of environmental degradation.

Developing Key Evaluation Questions

Key Evaluation Questions (KEQs) may be included in the Terms of Reference for an evaluation or may be developed by evaluators as part of the task of framing an evaluation. Rather than adding a separate question about environmental considerations the KEQs can integrate consideration of natural systems into all of the evaluation questions. A generic set of KEQs that integrate environmental considerations has been developed by the footprint evaluation project by Jane Davidson and Andy Rowe as a tool to prompt thinking and discussions between evaluators and evaluation commissioners.

The questions are purposefully framed as generic questions that can be adapted to meet the needs of specific evaluations. While it may not be possible to include all of the questions in an evaluation, considering the KEQs will lead to informed decisions about what questions to include or exclude and why.

Reference existing environmental commitments

Existing environmental commitments may be international agreements or national or local policies or strategies. They may be included as part of addressing the OECD-DAC criteria of relevance of cohesiveness or as a stand-alone rationale for illustrating the importance of including environmental considerations in an evaluation.

A step by step guide to using the World Fact Book to discover what international agreements a country has signed up to is available here. Keyword searches of national and local policies, strategies and plans can help discover environmental commitments internal to a country or region.

Making the argument

Including environmental considerations in evaluation is important and urgent because threats to the environment and ultimately human health and wellbeing are urgent and important. There are extensive resources available to support arguments about the importance and urgency of the issue. Environmental sustainability is also an equity and human rights issue. People and nations with fewer resources will have less capacity to implement climate change mitigation strategies.

Other relevant approaches and interventions

Footprint evaluation complements other cross-cutting initiatives in evaluation which address related issues including:

Join the community

If you'd like to stay up to date with this project, sign up for the Footprint Evaluation newsletter.

Resources

Tools

OECD-DAC Criteria

The OECD/DAC Network on Development Evaluation has updated the Evaluation Criteria definitions and Principles for use after 25 years. The goal of the criteria is to support better evaluation that in turn leads to better policies to advance both the 2030 Agenda for Sustainable Development (including the Sustainable Development Goals) and the Paris Agreement within the United Nations Framework Convention on Climate Change.

Key Evaluation Questions

These generic key evaluation questions (KEQs) are designed to support the inclusion of environmental sustainability by embedding consideration of the environment in each evaluation question rather than adding environmental considerations as a standalone question.

Guide to using the World Factbook to find a country’s international environmental agreements

This is a simple step by step guide to using The World Factbook to find the international environmental agreements a country has signed up to.

Blog

Memorial Day Week: Sustainability-Ready Evaluation by Andy Rowe: In this AEA365 blog, Andy Rowe argues that human system interventions have direct effects on the natural system; ignoring these direct effects causes evaluation to have a systematic and positive bias.

Discussion papers

Sustainability‐Ready Evaluation: A Call to Action: To facilitate the development of sustainability‐ready evaluation, this paper by Andy Rowe provides an initial checklist and references to useful resources.

About the Footprint Evaluation Project

This thematic page is an output of the Footprint Evaluation project - a co-creation and research project supported by the Global Evaluation Initiative

You can find out more about the Footprint Evaluation Project here, including the project's learning questions and planned outputs.

This thematic page is a work in progress and we will be sharing more resources and examples here.  Please feel free to use the comments section below to add suggestions for resources or other organisations and events relevant to footprint evaluation.

A special thanks to this page's contributors
Author
Director, ARCeconomics.
Vancouver Island, Canada.
Author
Founder and Owner, Real Evaluation LLC (now based in Seattle).
Seattle, United States of America.
Author
CEO, BetterEvaluation.
Melbourne.
Author
South Africa.
Contributor
Research Fellow, RMIT University.
Melbourne.
Contributor
BetterEvaluation Knowledge Platform Manager, BetterEvaluation.
Melbourne, Australia.

Comments

There are currently no comments. Be the first to comment on this page!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.