Week 26: Weighing people’s values in evaluation

By
Laura Rodriguez Takeuchi

What is more important to you: a good education or a good healthcare system? Or perhaps employment or security is at the forefront of your mind at the moment. What about the environment or human rights?

We all have different priorities in life and different sets of values with which we make judgements on things around us. Evaluations attempting to understand effects on people’s lives should at least attempt to try to understand the values of those people rather than imposing an external set of values. This week’s guest blog is from Laura Rodriguez Takeuchi, a researcher at the Overseas Development Institute. She introduces some practical ways that evaluators can begin to weigh people’s values as they relate to desired outcomes and distribution of benefits.

One of the very first steps of for conducting an evaluation is to determine what ‘success’ looks like. BetterEvaluation suggests that there could be inherent success criteria (for example, explicit programme objectives or sector-wide standards) or success criteria that emerge from tacit values, and these often have to be negotiated among different groups of people.

The process of defining and agreeing the values which will be used to form the judgements is often one of the more mysterious and less systematic parts of evaluation. Moreover, the question of whose values matter is sometimes overlooked.

Many programme and policy evaluations attempt to understand the effects of interventions on people’s lives without first understanding what wellbeing and satisfaction really mean to those people. Wellbeing is not a straightforward concept to grasp, let alone measure. It is comprised of multiple aspects and although there is a growing consensus on the set of things that are important in people’s lives, there is less agreement on how to capture the relative importance of each one. A new ODI paper explores some of the alternatives to incorporate people’s values in an index of wellbeing, which could provide a useful tool for budget allocation and evaluation of interventions across different sectors.

In the absence of specific information about the values attached to different aspects of wellbeing, a commonly used approach is to set equal weights for various dimensions. This is the approach taken in the Human Development Index and the Multi-Dimensional Poverty Index, assuming that all domains – health, education, income/living standards - have equal importance on a person’s overall wellbeing. This can’t be assumed - it is a matter for empirical testing to determine whether this is true for different people (and some have found that it is not ).

What if we asked people instead and developed weights from their responses? This doesn’t just mean having a chat – it can and should be done in a rigorous, methodical way. There are a number of approaches for eliciting values and in the paper we review six of them currently used in the UK health sector, but which have potential to be applied in the context of broader development interventions (see table 1 and appendix 1 in the paper for more info on these):

  • Standard gamble
  • Time trade-off
  • Person trade-off
  • Discrete choice experiments
  • Rating scales
  • Swing weights

The general idea is to ask people about hypothetical case scenarios, and to derive from those responses the weights that reflect how people value different domains of wellbeing. For example, in a discrete choice experiment (DCE) respondents are asked to select one of two hypothetical persons with different poverty profiles which they would consider most worthy of public support (figure below). When a large number of these responses are gathered, for a combination of different hypothetical persons, it is possible to determine which of the domains of wellbeing people place a higher value on.

 

Answering these questions is not easy (try selecting a person from the example above), and tools need to be carefully developed, tested and adapted to different needs and contexts. The fact that the health sector already uses them quite widely, should encourage us to think that the methods can be expanded and applied to other areas of development.

A more important issue is to recognize that different people may hold different opinions, so arriving at a single set of values may not often be possible, and that the aim perhaps should be to determine a plausible range of values. It is nevertheless important to recognise the values of different groups of people, for example, donors and communities affected by a development intervention, or groups of specific beneficiaries of a programme (women, children, those living in a certain area, etc.), to be able to have a sense of how interventions may affect their wellbeing in different ways. Group-based methodologies for eliciting weights can also be used, but have their own set of difficulties, especially when power dynamics are involved.

If development is about making people’s lives better, we surely need to ask them first about what they value the most. There is wide range of methodologies available to do it in a rigorous way, and careful thought needs to be given to selecting an appropriate method that is feasible for the context in which it is to be applied. Other methodologies, which are not based on people’s responses, are still useful, and can be incorporated in other steps of the evaluation process, but the value question in evaluation cannot be ignored.