Evaluating sustainability: Evaluative support for managing processes in the public interest

This special issue of New Directions in Evaluation includes discussions of different types of sustainability – sustainable environment, sustainable development, sustainable programs, and sustainable evaluation systems – and a synthesis of these different issues and their implications for transforming evaluation in ways more appropriate for evaluating sustainability.

This resource and the following information was contributed by Patricia Rogers.

Note: Many of the papers in this issue are behind a paywall, however Patricia Rogers has done a detailed summary of the key points from a selection of these papers below.

Authors and their affiliation

George Julnes (Ed.)

Key features

New Directions in Evaluation is published quarterly on behalf of the American Evaluation Association by Wiley Subscription Services, Inc. Access to all issues is included with membership in the American Evaluation Association. Access to the issue or individual articles is available for a fee.

Details of papers and editor's notes by George Julnes:

1. Evaluating Sustainability: Controversies, Challenges, and Opportunities | George Julnes

Sustainability, long focused on overpopulation and resource shortages, is now broadly applied in evaluation. This chapter provides a foundation for the evolving understandings of “sustainability” for sustainable environment, development, programs, and evaluation systems.

2. Sustainability-Ready Evaluation: A Call to Action | Andy Rowe

Evaluation is at the cusp of two urgent challenges: indigenous evaluation and sustainability. How we respond to these challenges can dramatically affect the future of evaluation.

3. Sustainable Development Evaluation: Understanding the Nexus of Natural and Human Systems | Juha I. Uitto

Sustainable development has three interlinked dimensions: social,economic, and environmental. Redressing the neglect of the environmental dimension requires approaches and skillsets that encompass both the natural and human domains.

4. Improving the Match Between Sustainability Questions and Evaluation Practice: Some Reflections and a Checklist | Eleanor Chelimsky

Sustained programs require continued funding but also remaining aligned with multiple public interest values. This calls for a new evaluator task of assessing likely changes in the relevant public-interest values as they relate to program or policy sustainability.

5. Achieving Sustainability Through Sustainable Organizational Evaluation Systems | Stephen Porter, Penny Hawkins

Organizational evaluation systems become sustainable through: being networked, being use focused, and undertaking processes to agreed quality standards. This is important in that sustained evaluation can more effectively contribute to the aims and values of sustainable development.

6. Transformation to Global Sustainability: Implications for Evaluation and Evaluators | Michael Quinn Patton

Whereas sustainability in evaluation has traditionally been associated with maintaining programs and their results, the infusion of systems thinking and complexity theory into evaluation transforms sustainability to focus on the whole Earth, and does so with a sense of urgency.

7. Supporting Transitions to Sustainability: Evaluation for Managing Processes in the Public Interest | George Julnes

Addressing sustainability in evaluation requires supporting processes, particularly the complex task of supporting societal transitions to sustainability. This requires better use of a “requisite variety” of multiple methods, strategies, evaluator roles, and worldviews in an effort to serve the public interest.

How have you used or intend on using this resource?

I've used this issue to better understand some of the issues related to evaluating environmental sustainability. Of particular relevance for evaluating environmental sustainability are the following papers:

1. Evaluating Sustainability: Controversies, Challenges, and Opportunities George Julnes

This paper provides a historical review of concerns about sustainability dating from at least 1713, including publications in the 20th century including Our Plundered Planet, The Population Bomb, and Silent Spring and initiative such as the 1970 USA National Environmental Policy Act, requiring environmental impact statements, the 1972 UN Conference on the Human Environment, the 1987 UN World Commission on the Environment and Development WCED (sometimes referred to as the “Brundtland Commission”), the 1992 UN Conference on Environment and Development (sometimes referred to as the “Earth Summit”), the 2000 UN Millennium Declaration and 8 Millennium Development Goals (MDGs), the 2002 World Summit on Sustainable Development, the 2003 UK Policy document “Sustainable Communities: Building for the Future”, and the 2015 UN 2030 Agenda for Sustainable Development and 17 Sustainable Development Goals (SDGs).

This review includes a summary of some approaches to thinking about sustainability.

The “three Es of sustainability” ( or three pillars) were described in the 1987 WCED – economy, equality, and environment. More recently these have been expanded to four or five pillars: environment, economy, social, institutional, cultural.

The “triple bottom line” developed by John Elkington built on these pillars to argue that businesses should report on their achievements in terms of profits (sustained financial profit/shareholder value), people (sustained human and social capital) and planet (sustained environmental capital). These three types of capital were seen as intrinsically important and also important for the other types of capital. The “five capitals model’ promoted by the Forum for the Future expanded this to five types of capital: natural, human, social, manufactured and financial capital.

The paper also discusses some particular challenges in evaluation. It briefly refers to the challenges of understanding processes given the complexity of the interacting systems involved.

It discusses in some detail the major challenge of how to think through and manage trade-offs between the three goals of economic profits, environmental protection and restoration and social equality and cohesion.

One paradigm, sometimes called “shallow greening,” prioritizes economic development but tries to achieve this with minimal environmental and social degradation; this is embraced by those who prioritize currently identified human needs and view the environments as a “bundle of processes, all of which have natural functions that serve the interests of human survival and contentment” (O’Riordan, 1992, quoted by Raco, 2005, p. 326). The competing paradigm, “deep greening,” reverses the priority, seeking to strengthen environmental protection and social cohesion while still respecting the importance of economic profits; in this view, sustainable development “is a never-ending process of progressive social change” (Kemp et al., 2007, p. 1). (pp25-26)

2. Sustainability-Ready Evaluation: A Call to Action Andy Rowe

This paper argues why evaluation needs to change to be sustainability-ready and how this can be done.

It begins by setting out the arguments why environmental sustainability needs to be included in all evaluations, given the importance of environmental conditions on the needs, effectiveness and durability of human service interventions including health, education, disability services, housing, transportation, public safety and economic and community development. It also shows how human service interventions and policies can have significant environmental impacts, using the example of policies about school siting and construction in the USA which have adverse effects on natural systems including the reduction of ecosystem services provided by the land (such as water quality management and biodiversity), increased CO2 emissions by increased private transport use, and so on.

The paper found very little work on this had been done in the field of evaluation, as evidenced by items in the AEA publications New Directions in Evaluation and the American Journal of Evaluation and pointed to other areas where work on evaluating environmental sustainability was occurring, including the Collaboration for Environmental Evidence (CEE), the Stockholm Environment Institute, and the International Institute for Environment and Development (IIED).

The paper sets out the beginnings of a checklist for sustainability-ready evaluation:

1. Assume that the evaluation needs to consider both social and natural systems – unless inquiry (such as in an evaluability assessment) says otherwise.
2. Open up the theory of change to create an ‘unconstrained theory of change’ ‘that extends the temporal and spatial reach of the intervention, articulating natural system as well as human system inputs, assumptions, mechanisms, and effects Caballero (2015), Sustainability Consortium 2017’.
3. Expand the framing of the evaluation to include consideration of important unintended impacts on natural systems.
4. Engage people with expertise in natural systems evaluations as core members of the evaluation team from the beginning and throughout the evaluation.
5. Identify and incorporate different scales and units of account for natural and human systems
6. Identify and include key stakeholders for all interests who can affect the intervention and who are affected by it – including representatives from natural systems.
7. Engage knowledge users in the process of developing knowledge to increase its relevance and legitimacy.

3. Sustainable Development Evaluation: Understanding the Nexus of Natural and Human Systems, Juha I. Uitto

This chapter provides an informed and succinct description of the different environmental threats which pose fundamental challenges to the ecosystems that support human life and societies.   

It discusses the need to identify and understand the nexus of natural and human systems in order to understand and achieve sustainable development:

Sustainable development occurs in the nexus where human (social and economic) and natural (environmental) systems meet. There are often trade-offs and conflicting interests between the two. Such trade-offs need to be recognized and carefully managed in policymaking and in designing and implementing development programs and projects (Puri, 2017). Evaluations must bring these trade-offs to the forefront. (p.55)

The chapter goes on to describe the evaluations of environmental programs and projects in developing countries carried out by the Global Environment Facility Independent Evaluation Office.

These show the need to understand the complex dynamic systems in which the interventions operate and interact, including unintended effects as well as stated objectives, and using mixed methods implemented by multidisciplinary teams.

The paper acknowledges both the usefulness and risks of quantifying the benefits for human systems of protecting and restoring natural systems:

“Environmental economics is rightly receiving increased attention, as ecosystems provide tangible goods and services to humankind in terms of food, water, materials, climate regulation, pollination, protection from disasters, and many others. It is nevertheless important to recognize that while the ecosystem services that nature provides to society have a value that can be monetized, in most cases man-made capital cannot substitute for natural capital and so natural capital should be maintained for its own value (Daly, 2005). Furthermore, from an ethical point of view, nature—both living and inanimate—has an intrinsic value that goes beyond its instrumental value (Washington, Taylor, Kopnina, Cryer, & Piccolo, 2017). This includes its cultural and spiritual value. Still, many decision-makers both in the public and private sectors appear to believe that what cannot directly be measured in monetary terms is not worth anything.” (p.50)

4. Transformation to Global Sustainability: Implications for Evaluation and Evaluators, Michael Quinn Patton

In this chapter, Michael Patton urges evaluators and others involved in evaluation to engage in transformation of evaluation practice to address the environmental crises.

The future sustainability of the Earth and humanity has reached a crisis level, but having acknowledged that, people go on about their business and hope that others deal with the crisis. This chapter argues that the crisis of planetary sustainability demands transformation of all economic, institutional, societal, and ecological systems together, that evaluating such transformation requires transformation of evaluation, and that failure to seriously address global transformation in everything we do as evaluators makes us part of the problem, not part of the solution. (p.105)

He quotes a keynote address on One Planet Transformation by Terry A’Hearn, CEO, Scottish Environmental Protection Agency at the University of Dundee (August, 2017) which spells out clearly the issue:

For the world as a whole, we are consuming the Earth’s resources as if we have 1.7 planets. We only have one. We are all heading in the wrong direction. There isn’t a single country that has reversed its over-use of the planet yet. Incremental change isn’t enough. We need transformational change. We need to transform our society and world to one that lives within what one planet can support. (A’Hearn, 2017) (p. 104).

Traditional linear planning of individual projects and programs for incremental change will not be enough – a major challenge for the theory and practice of evaluation which still focuses primarily on individual projects and programs and achievement of their stated objectives.  Transformative interventions require rethinking how change occurs and how evaluation can support this:

Transformational initiatives must be multifaceted, multidimensional, multisectoral, multinational, and multiplicative. Transformation flows from an understanding that the status quo is not a viable path forward and that action on multiple fronts using multiple change strategies across multiple landscapes will be needed to overcome the resistance from those who benefit from the status quo. Multiple interventions are needed to multiply effects, creating streams of diverse interventions flowing together to generate mammoth change in global systems. Thus, transformation is simultaneously and interactively global and local at the same time, contextually sensitive and rooted while being globally manifest and sustainable. Tracking these new, transformational initiatives will require a complex global systems change approach to evaluation meaning that evaluation must be developed and adapted if it is to be part of the solution rather than part of the problem.” (p.112)

The chapter sets out 4 propositions:

“Transformation Principle 1: Global–Local Dynamic Interconnection ​Principle.

Connect global and local perspectives, knowledge, and understandings in support of change. Global systems change must be contextually sensitive and grounded in the interactions between local and global processes and scales of change. The term that captures this is GLOCAL (Steger, 2013, p. 2). For evaluation, this means applying a multilevel connectivity criterion: Assess global–local interactions and interconnections. This likely will involve documenting contextual variations locally within a coherent global pattern of transformation. …

Transformation Principle 2: Cross-Sector Principle.

Integrate and coordinate interventions across sectors and traditional program areas (cutting across silos). Transformational interventions work across sector divisions and program specializations. The first principle (#1 above) concerns the global focus of an intervention. This second principle concerns cutting across program silos to achieve integration and synthesis. Part of the barrier of model project thinking is that projects and programs tend to be narrowly siloed. As closed systems with specific, limited, and delimited targeted outcomes, the very effectiveness of projects and programs is predicated on their narrow focus. …

Transformation Principle 3: Multiple Intervention Strategies Principle.

Target mixed and multiple types of changes. Transformation requires multiple interacting strategies on multiple fronts: regulatory, policy, incentives, education, organizing, services crossing social, economic, and environmental arenas. For evaluators, this means applying a strategic integration criterion: Track and analyze the interactions and synergies of multiple and diverse interventions and initiatives. …

Transformation Principle 4: Design and Implement Transformation-Aspiring Interventions for Adaptive Resilience.

Sustainability as adaptive resilience is dynamic, complex, and developmental in formulation and evaluation. For evaluators, this points to a resilient sustainability criterion: Assess sustainability over time for adaptive resilience as dynamic, complex, and developmental in formulation and evaluation.” (pp113-115)