Week 43: Evaluating sustainability


Meeting sustainability goals is one of the most important and urgent challenges for humanity yet for many of us, thinking about sustainability in evaluation means assessing the extent to which the benefits of a project, programme or policy are maintained after formal support has ended.

In this week’s blog we see another view of sustainability, one which will help evaluators and commissioners of evaluation to consider the impact of the project (or other forms of intervention) on the sustainability and sustainable development  – with the latter defined as ‘Development that meets the needs of the present without compromising the ability of future generations to meet their own needs’ (World Commission on Environment and Development 1987). Sustainability and sustainable development were given high prominence in the late eighties and early nineties, and been the basis for many of the goals and objectives of UN and other organsiations.  The science and practice of operationalising the concept of sustainability remains challenging, and continues to be developed. Deborah O’Connell shares some reflections from a recent report that she and her colleagues from the Commonwealth Scientific and Industrial Research Organisation prepared for the World Economic Forum last year.

Our work on sustainability at CSIRO grew from our research into the biofuels sector, which started at a time when food versus fuel was making headlines around the world in 2007. Exploring the concept of ‘sustainable biofuels ‘ required gaining an international perspective on not just the food versus fuel dimension of sustainability, but a range of different sustainability aspects, including the carbon footprint, the water use, the clearing of forests and impacts on biodiversity, and the social issues for direct and indirect effects.

We started working with the World Economic Forum Agenda Council on a review of sustainability theory and implementation over the last couple of decades. The lively discussions across sectors in this forum helped to crystallise the learning from our other research. This work on Navigating Sustainability was published by the World Economic Forum as a Summary Report, and the Full Report is published by CSIRO.

There are many learnings from this review, but I put forward three which have specific relevance to the BetterEvaluation audience.

There are many approaches to measuring and evaluating sustainability, and the field is evolving rapidly

Consensus is emerging that characterising the functioning of the physical, ecological, and social systems which support human life and understanding the interactions between these systems are fundamental. Different approaches sustainability have different strengths, weaknesses, and applications. The diversity of these approaches can be very confusing. There are many parallels with the blog posted by Patricia Rogers in week 19 about the ways of framing the difference between research and evaluation. She said that it was challenging that important terms like ‘evaluation’, ‘indicators’, ‘monitoring’ being used in very different ways by different people. Likewise, sustainable development and sustainability mean different things to different people, and vary across different contexts.

This means that creating workable operational tools for assessing and promoting sustainability will always require judgments that are subject to legitimate debate. Even with widely accepted definitions for sustainability and sustainable development, the central questions of ‘sustaining what, for whom, where, and for how long?’ remain laden with human values and social choices. These values and choices are very context-specific and therefore differ across time, space, and culture.

I imagine that questions and challenges resonate with the Better Evaluation audiences, particularly those required to use criteria such as those produced by the OECD Development Assistance Committee – that mention sustainability but include no standards as to how to apply this in a particular case.

Data and measurement are crucial, but are insufficient alone for evaluating sustainability and enhancing sustainable outcomes

Within the world of sustainability theory and practice, there is a very high emphasis on data and measurement and monitoring - particularly of a burgeoning number of complex indicators (see Full Report for further discussion). The evaluation component of sustainability is often implicit to particular theoretical construct, and the lack of clarity on this element is one of the reasons for confusion about different approaches, as well is the debate about how to measure or claim ‘success’.

Within the Better Evaluation audience, there is a clear emphasis on using explicit stated evaluation methods to interpret and derive meaning and understanding from data.

There is no doubt that measuring sustainability will be most effective when measurement and evaluation are understood as part of a wider iterative process of learning and acting, and embedded in institutions that can provide the mechanisms for implementation.

In reviewing the multiple approaches to sustainability theory and practice, we identified four universal building blocks as essential, no matter which definition, theoretical construct or method is used (Figure 1) :

Institutional mechanisms: the formal and informal rules that provide the governance, oversight and stability necessary to implement the sustainability framework

Data – specification, collection, analysis and the use of projections: data, which can consist of measurements, modelled interpolations or projections, are used as the basis for evaluation

Evaluation: interpreting the meaning or value of the data in relation to agreed sustainability objectives

Feedbacks: flow of information or action between components of the framework, including catalysing changes that promote sustainability.

Figure 1.

The four elements of a robust sustainability framework

To be effective in achieving sustainability, these four elements need to be addressed and linked across a range of scales (Figure 2). Greater availability of data will only be effective for sustainability if it is used within such an approach.

Linked activities in each of the four elements thus need to be integrated across different scales (Figure 2). Individual enterprises, each sector of a national economy, and each country can aim for sustainability in relation to local goals, and contribute to sustainability at the next scale up or down (eg regionally, nationally, globally). The framework proposes that effective activity in each element, at each scale, and effective links across scales are required in order to achieve global sustainability outcomes. Activity in any one of the elements can make a partial contribution towards global sustainability outcomes.

Figure 2.

Feedback mechanisms and interactions across scales in a robust sustainability framework

Many examples are provided in the Full Report demonstrating how these linkages occur, with particular focus on some industry sectors such as forestry, bioenergy, and extractive industries.  In our review of approaches to date, it is clear that there is a great deal more transparency and clarity around the elements of DATA: COLLECTION, ANALYSIS, PROJECTIONS and INSTITUTIONAL MECHANISMS, at least within formal settings. There is a lack of clarity and understanding of the implicit methods that are often deployed in the EVALUATION element, and the FEEDBACK element is frequently absent. The use of this framework linking across scales has been a useful way to provide the basis for discussion of the complex topic of sustainability in a range of project-related settings: for example, endless committee discussions about the utility of various indicators can frequently be circumvented by showing clearly at which scales, and within which evaluation constructs different indicators may be applicable.

We need to develop guidance to simplify choosing the right methods and approaches for different purposes, and to make their use more effective in promoting sustainability

Meeting sustainability goals is one of the most important and urgent challenges for humanity. Human pressures already exceed the safe coping capacity of the planet in some issues. In some cases, such as greenhouse emissions, we have only one or two decades to achieve a substantial change in trajectory – or risk extreme impacts.

Sustainability assessment tools have a crucial role in informing, guiding and motivating this action – and are already making an important contribution.

M&E methods also play a critical role. As well as data collection and analysis methods, the BetterEvaluation Rainbow Framework offers a whole list of tools and strategies for developing evaluative criteria which can embed sustainability requirements from different perspectives. The list includes formal statements of values, ways of articulating and documenting tacit values, and strategies for negotiating between different values.

Existing tools are not perfect, and will continue to be refined. New tools and approaches will need to be developed and implemented, addressing gaps and providing new traction and value. Technical, economic and social challenges will need to be addressed and overcome. A growing number of public and private enterprises will engage in different ways, for different reasons, around different aspects of sustainability. To this end, we are currently conducting a series of in-depth interviews with sustainability influencers and practitioners in a range of organisations – including those that fund or manage projects in developing nations; large and small enterprises in manufacturing and technology; researchers and policy-makers – to better gauge their knowledge needs in the area of sustainability implementation. We hope to build an improved understanding of the decisions that they make with respect to sustainability, the theoretical constructs and evaluation methods that they already use, and what knowledge they need in order to better support such decisions into the future.

Image source: Solomon Karmel-Shann (image produced for the cover of the Full Report)