Key Evaluation Questions (KEQs) to guide Footprint Evaluations

The key evaluation questions (KEQs) are designed to support the inclusion of environmental sustainability by embedding consideration of the environment in each evaluation question rather than adding environmental considerations as a standalone question.

This resource and the following information was contributed by Kaye Stevens.

This resource is a draft and the authors would appreciate any feedback or suggestions about how to make this resource useful and relevant. You can join the Footprint Evaluation Community to give feedback on this and other outputs of the Footprint Evaluation Project.

Authors and their affiliation

Jane Davidson and Andy Rowe

Year of publication

2021

Type of resource

  • Tool

Key features

The list of KEQs is an output of the Footprint Evaluation project. It is grounded in a conceptualization of evaluation as being fundamentally about asking and answering evaluative questions. The KEQs build on earlier versions of generic KEQs in Jane Davidson’s evaluation workshops and publications.

As stated in the introduction: “Identifying the relevant values and unpacking the evaluative terms in each of the KEQs is a core part of the job of answering them. This is no simple or formulaic task; these discussions are an extremely important part of any evaluation. …. This list of KEQs has been designed so that it can apply in any sector, type of evaluand, level of analysis, etc. As such, the list is deliberately generic; each evaluation team should rewrite/interpret the questions for the particular sector, context, culture, population/community, evaluand, and evaluation audience, using wording that makes sense for that application.”

Who is this resource useful for?

  • Commissioners/managers of evaluation;
  • Evaluators;

How have you used or intend on using this resource?

The KEQs can be used in conjunction with, or as a way of applying national or organizational evaluation criteria such as the OECD/DAC evaluation criteria.

Why would you recommend it to other people?

The KEQs are valuable prompts for discussing how to embed consideration of the natural system in the evaluation of interventions primarily concerned with human systems.

1
Average: 1 (1 vote)
Rate this Resource:

Comments

David Roberts's picture
David Roberts
Rating: 
0

The avowed purpose of the paper is to embed sustainability into an evaluation; a laudable goal. It is also  suggested that the paper can help people apply the DAC criteria. 

Sustainability in the DAC criteria relates to the "extent to which the net benefits of the intervention will continue."  The discussion on sustainability within the DAC criteria places an emphasis on equity as a condition of sustainability.  The authors of the paper do not explicitly state that the KEQs might embed equity issues into an evaluation.  Indeed they may not intend that the KEQs be used for that purpose. However, it has been suggested by others that the KEQs outlined in this paper can be used to embed equity in evaluation.

There are a number of contradictions inherent in this paper that mean it does not achieve the second purpose attributed to the paper. I will touch on a few of the issues here.

First, it is useful to have a check list of things to look for in an evaluation.  The authors also acknowledge that any KEQs need to be operationalised for any evaluation. However, presenting a list of standardised KEQs implies standards that should be applicable to all evaluations.

Such a standards-based approach (with thanks to Stake for the concept), particularly given the way the Introduction gives prominence to assigning goodness, value etc. to the evaluand, implies there is a 'true' standard against which such values might be measured. A standards-based approach is inherent in the notion of KEQs. 

However, such an approach necessarily privileges one set of values and does not reflect the reality of many different and conflicting sets of values. The choice of value sets could still be equity-based, if limited, but the language of both the Introduction and the KEQs themselves strongly suggests that the values to be privileged are those of either the evaluator or the commissioner. In other words the values being privileged are those of the people "doing" things to others.

KEQ4: Outcomes & Impacts is all about the contribution of the evaluand. It is framed as contributions to changes in the presenting situation but the focus is on what the evaluand does, without any explicit reference to the external factors, circumstances and people within which the evaluand operates and with whom it interacts. Nor is there any explicit reference in this KEQ to differentials in the impacts or how the  stakeholders interact with the evaluand or how it accords with their values and desires.  There is a passing reference to the possibility of "adverse impacts" but no systematic discussion of how one might use KEQs to systematically discover unintended consequences. Indeed I can't see how these KEQs could lead to a systematic exploration of unintended consequences.

KEQ5: Patterns, Outliers and Links does reference  the potential variability of impact, which is good. However, the recognition of variability is framed in terms of program performance not in terms of the impacts on and responses of the various stakeholders; and self-determination is notably absent from the discussion on outcomes and impacts. 

There is a reference to self-determination in KEQ3: Implementation, but not to how self-determination might influence how an evaluation is conducted or what conclusions might be drawn. And again the language is all about what is DONE by the evaluand, not about self-determination. The introduction of the concept of implementation fidelity here is  symptomatic.  Implementation fidelity is a concept that assumes the evaluand is a machine-like phenomenon. Something that can be designed and delivered as designed with limited variation; rather than an interaction between different and variable human beings with notion roles of "deliverer" and "recipient."

Overall, while the KEQs might help embed considerations of some notions of sustainability, it does not, and cannot, embed equity into evaluations.

E. Jane Davidson Ph.D.'s picture
E. Jane Davidson
Rating: 
0

Kia ora David,

Thank you so much for taking the time to review this paper and provide detailed comments. At this point it is a draft document that we hope to improve on substantially, so your critique will be very helpful for shaping the next iteration.

We are indeed very keen to ensure that equity considerations are infused throughout the KEQs, just as we are seeking to infuse environmental sustainability considerations. As you rightly note, that's not easy to do, particularly getting the right balance between depth and breadth of explanation while still keeping the KEQs as succinct and user-friendly as possible. 

One of our goals with the Footprint Evaluation Project is to explore a collaborative approach to creating and sharing knowledge about better evaluation practice. To this end, we have set up a Footprint Evaluation Community where interested participants from the wider evaluation, natural sciences and sustainability communities can help us puzzle together about how to make resources like this most useful and relevant - and to identify what else is needed to help power this movement. 

We would be delighted if you (and anyone else reading this who is interested) would be willing to join us in that discussion and collaborative knowledge creation. If you follow the link to the Footprint Evaluation Community, you will see a sign-up option. 

We will be posting something soon introducing the draft KEQs and inviting thoughts and suggestions. Andy and I are very much looking forward to learning from and with the Community - and of course producing a better next iteration that we hope some of the Community (both commissioners and evaluators) will be willing to field test the KEQs and help us take them to the next level. 

Again, thanks so much for your interest and thoughtful comments! 

Jane

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.