What is evaluation to do? Ways of responding to the climate and environment crisis – Part 1

This blog is the first of a 2-part series on the issues raised about COP26 in papers published in the journal Evaluation.

It is sometimes hard to not be discouraged about the power of evaluation to make a difference, given the latest IPCC report and escalating environmental and humanitarian crises. But it is essential to hold onto hope and to work for change. A recent publication offers some timely and valuable suggestions for what can be done by those involved in evaluation – including evaluators, evaluation managers and evaluation educators.

After the COP 26 meetings in Glasgow last year, the editor of the international journal Evaluation, Elliot Stern, invited 12 papers on the lessons we might take away for evaluation. (Full list of contributors at the end of this blog, including the late Eleanor Chelimsky, whose work has been showcased in a recent video by Michael Quinn Patton). As part of this, I wrote a paper about needing to expand the focus of what is evaluated beyond intended outcomes and discrete projects. Only when the issue was published did I see the scope and richness of the other contributions, which present differing and complementary views and advice. The key messages from the papers are worth pondering – they all require changes in how evaluation is practised, the skills that evaluation teams and managers have, and how evaluators and evaluation managers are trained and supported.

The combined papers are available as Open Access, so I urge you to read the detailed versions. Here are the key messages as I see them.

1. The situation is dire

Several writers reminded us what is at stake:

“The effects of inaction on climate change and ecosystem breakdown are already taking their toll on those who are least responsible. Continuing to postpone immediate and necessary action maintains humanity’s inevitable and irreversible path towards rapidly increasing global suffering”. (Ian Davies)

“Increased GHG emissions have made natural disasters more frequent and severe. Beyond a tipping point, they could well trigger irreversible ecosystem collapses.” (Robert Picciotto)

The situation is clearly shown in the Climate Action Tracker model (referred to by Robert Picciotto’s paper), which shows the shortfall in current policies and actions – and even in current pledges and targets. I recommend checking this out to see its summary of your country’s current policies.

The COP26 meeting might best be described as a half-full glass, with some progress and commitments but falling well short of what was needed. Its final statement, for example, referred to “phasing down” coal-fired power rather than “phasing out”, and the pledged actions are still insufficient to avoid a 1.5-degree increase in average global temperatures with the negative impacts on people and the planet that this will bring.

2. Evaluators need to be part of the solution

For the first time, the COP26 meeting included the adoption of a new work program on the global goal of adaptation that includes an objective to ‘Facilitate the establishment of robust, nationally appropriate systems for monitoring and evaluating adaptation actions’.

In addition to these specific projects, there is a need for all evaluators and evaluation managers to include climate implications in evaluations, given the impacts of broader projects, programs and policies.

In his paper, Rob van den Berg drew on his previous experience as Director of the Independent Evaluation Office of the Global Environment Facility to discuss what he refers to as the micro-macro paradox of climate action:

“success and effectiveness at the micro level of interventions, yet no visible change … at the macro level – an ever-fastening slide of the world towards a climate catastrophe through ever increasing levels of greenhouse gas emissions”

due to the fact that

“climate action funding was outpaced by public funding of practices that caused climate change ... to the tune of 1-100”.

Rob went on to urge evaluations to include consideration of these impacts

"Evaluations could even be mandated by the UN and governments to identify in future scenarios the potential damage that ‘business as usual’ interventions and policies can cause to humankind and its future on planet Earth.”

Ian Davies did not mince words, making it clear that he sees current evaluation practice as complicit:

“… evaluation as it is framed and practised appears at best as a trompe l’oeil for effective global action and at worst as a willing foot soldier of ongoing colonial dominion”.

Scott Chaplowe called for evaluation to “take a public position on climate change and related complex environmental and social issues”. Public positions are already seen in other sectors and organisations – for example, a joint statement by over 200 health journals “calling on leaders to take emergency action on climate change to protect health”.

3. Equity needs to be part of this

Equity and environmental concerns are intertwined, not competing, concerns, as noted in several papers.

"Humanity’s catastrophic situation is rooted in pervasive injustice and inequity, disregard for human rights and the continued and unsustainable overexploitation of our ecosystem by a political and economic system developed by the haves, at the expense of the have-nots.” (Ian Davies)

“…there will be no ecological transition if injustices are not addressed concurrently. Not only environmental imbalances and depletion increase social inequities, but they are largely intertwined in a long-standing system of domination and exploitation of human and natural resources.” (Astrid Brouselle)

Therefore, efforts to address environmental concerns need to connect with and contribute to existing efforts and public statements to address equity concerns in evaluation. This includes engaging rights holders in evaluation processes and ensuring their concerns are adequately represented.

“[Evaluators need] to support wider engagement – to contribute to creating space for dialogue which are more inclusive, especially for those who are marginalised and those with different cultures and world views, and to reflect these in broader theories of change, which show interventions, environmental and social impacts in the short and long term, intergenerationally, locally and globally.” (Astrid Brouselle)

4. Follow the money

Both Eleanor Chelimsky and Jindra Čekan drew attention to the need to focus on money. For climate change interventions, it will be necessary to look critically at what pledges have been made, what money has been committed, and what this money was spent on. There has to date been a significant shortfall between promises and action.

In addition, expenditure on actions with significant climate change implications must be considered, when these are not identified as climate change interventions.

In my own country, I can point to the situation in Australia where fossil fuel subsidies cost $11.6 billion in 2021-22, an increase of $1.3 billion in the last year[1], massively outweighing investment and impact of climate change mitigation projects. This analysis has not been done by government-commissioned evaluations but by an independent think tank – pointing to the independent evaluations needed for real accountability (more on that later).

This raises a question for me as to whether evaluators or evaluation teams currently have the appropriate expertise to design and implement evaluations that can critically examine claims about program expenditure and track where money has gone.

5. Pay serious attention to unintended impacts

As well as looking at where the money goes, evaluation needs to look seriously beyond the intended outcomes and impacts of interventions. This includes identifying shifting risks and negative impacts, as pointed out by Timo Leiter, Dennis Bours and Debbie Menezes in their paper:

“successful adaptation in one community, region or system can occur at the expense of another, causing unintended consequences known as ‘maladaptation’”

It is 50 years since Michael Scriven first advocated for ‘goal-free evaluation’[2] – evaluation which looks at actual effects regardless of whether these were the stated goals – but evaluation practice and commissioning practice still often focus primarily or exclusively on the achievement of stated objectives.

In his paper, Andy Rowe urged relabelling predictable negative impacts as ‘ignored’ rather than ‘unintended’ and argues strongly that these need to be included to ensure valid evaluations:

The value ascribed to interventions will usually overstate its true value because evaluation ignores natural systems. It is not a stretch from that to suggest that our contributions are deepening the sustainability crises. The interventions cause the harm, and evaluation allows that harm to continue. … [A] do-no-harm standard when human and natural systems are hardwired necessitates evaluations providing an explicit assessment of the effects of the intervention on natural systems. All evaluations. Failing to do so is to presume that it is OK for your interventions to continue to cause environmental harm.

[1] Source: https://australiainstitute.org.au/report/fossil-fuel-subsidies-in-australia-2021-22/

[2] Scriven, M. (1972). Pros and cons about goal-free evaluation. Journal of Educational Evaluation, 3(4), 1-7.

Contributors to the special issue

• Rob D. van den Berg, past President of International Development Evaluation Association (IDEAS), former Director of the Independent Evaluation Office of the Global Environment Facility
• Dennis Bours , coordinator of the Adaptation Fund: Technical Evaluation Reference Group (AF-TERG) secretariat
• Astrid Brousselle, professor and director of the School of Public Administration at the University of Victoria, Canada
• Jindra M. Čekan, political economist who works in international development
• Scott G. Chaplowe, evaluation and strategy specialist
• Eleanor Chelimsky, independent consultant  and former director of the US General Accounting Office
• Ian C. Davies, evaluation consultant, past President of EES
• Weronika B. Felcis,  Latvia University, Board member of EES and Evalpartners
• Timo Leiter, independent consultant on climate change adaptation and monitoring and evaluation
• Debbie Menezes, chair of the Adaptation Fund: Technical Evaluation Reference Group
• Robert Picciotto, former Vice President and Director General of the Independent Evaluation Group of the World Bank, and a founding board member of the International Evaluation Academy
• Patricia J. Rogers, consultant, Member of the Footprint Evaluation Initiative;
• Andy Rowe, evaluation consultant, former President and a Fellow of the Canadian Evaluation Society
• Juha I. Uitto, Director of the Global Environment Facility Independent Evaluation Office.