52 weeks of BetterEvaluation: Week 42: Using value for money in evaluation - a conversation
"Value for Money" is a term that is increasingly used in evaluation - often to mean very different things.
This week we're delighted to launch a new paper on Value for Money, written by Farida Fleming, an evaluator from Assai Consult and PhD Candidate at RMIT University, and developed by a Working Group convened through the Australasian Evaluation Society. In this week's blog post, Farida discusses Value for Money with Julian King a Public Policy Consultant from Julian King and Associates Ltd.
Farida: "Value for Money is a topic that is high on the development agenda. But a lot of practitioners are confused about what VfM means. What seems to be agreed is that VfM is based on assessments of economy, efficiency and effectiveness, and some add in equity as a VfM consideration."
Julian: "Although there is no universally agreed definition, it may be useful to offer one here. I often paraphrase Drummond, O’Brien, Stoddart & Torrance 1995, Methods for the Economic Evaluation of Health Care Programmes to say VfM is a measure of whether something is worth doing, bearing in mind its performance, its costs and alternative use of resources.
In simpler language, VfM boils down to 'using resources well'. When we ask whether something provides VfM, we are asking "is it good enough to justify the resources used?"
Farida: "I think coming to a simple definition is useful. But does this definition apply equally across all parts of the project cycle? Although DFID provides some useful guidance on how VfM can be applied at all stages of a project, how do we properly account for the differences of perspectives of practitioners who are only involved in one or a few parts of the cycle?
Before we agree on a consensus definition of VfM, it may be useful to explore some contentious issues. For example, some say that VfM is a concern for donors rather than beneficiaries. Taking this perspective, VfM is a step backwards. It moves away from international development agreements from the Paris Accord onwards that aid is and should be country led - for an interesting perspective on this, see Fred Carden's short piece ‘Whose Development Results Count?’, in NORRAG NEWS."
Julian: "Fred Carden raises some very important points, but these are criticisms of how VfM has often been evaluated, rather than the validity of VfM itself. If donors have limited resources, wouldn’t it be in beneficiaries’ interests that those resources are allocated where they can make the greatest difference? Perhaps their problem isn’t really with the concept of VfM but with the criteria used to evaluate it."
Farida: “I think the critique has to do with the money and resources required to determine VfM. This can be seen as yet another trend generated by donor countries whose implementation consequences are borne by developing country colleagues.
But yes, the other point you raise is another critique of VfM - that it can lead to a focus on activities that are easy to measure and implement rather than more risky and potentially more important activities. Exploring these differences can lead to agencies developing their own position on what VfM means for them. For example, for Christian Aid, VfM focuses on Effectiveness and Equity, with Economy and Efficiency only relevant in terms of understanding the other two issues.
But I think we need to come back to the practitioner in all of this. Do you think that there are situations where VfM assessments are useful and when they are not?”
Julian: “My own view is that the concept of using resources well is always relevant (though of course it may assume a higher priority in some contexts than others). How useful a VfM assessment may be is related to how, rather than whether, it is done: you need to pick the right criteria and methods to be fit for purpose.
For example, economic methods for evaluating VfM, (such as cost benefit analysis and cost-effectiveness analysis) which reflect a worldview in which social welfare is maximised through efficient allocation of resources, can limit our ability to assess VfM. If we know there are values in society that are not well represented by summing individual values, then an economic method might reinforce power imbalances and discriminate against minority groups. When evaluating VfM in social or development investments, issues such as equity and justice may be just as important, or more important, than efficiency.”
Farida: “Yes, with more and more aid being focused on fragile states, addressing equity can be really important. In this type of country context, it may not be enough to have an average positive effect. It may be just as important to not have a negative effect on the most disadvantaged.
A good example comes from Gill Westhorp's work on parenting programs. She found that parenting programs were having good outcomes for the majority of participants. But for the most vulnerable 10% of the parenting population, which mainstream services were failing to reach, the program was having a negative effect. So we need to consider the impact (and current and ongoing costs) of alienating the 10% of the population with the highest needs. ”
Julian: “Another problem in cost benefit analysis is the treatment of "intangibles". In theory, anything that has utility to people can be valued in monetary terms, however, there can be ethical, conceptual and practical difficulties in doing so. Things that have a high social value (such as human lives, cultural identity, relationships) are fundamentally priceless. In my work I define tangibles as "things that can and should be valued in dollars" and intangibles as "things that can't or shouldn't be valued in dollars".
Cost-effectiveness analysis enables intangibles to be measured in units that make sense (e.g. life years saved), but only if they can be represented in a single measure that is readily quantified. Often in social investments the most valuable things can be the hardest to measure.
For example, if a cost-effectiveness analysis says Medicine A saves more lives than Medicine B at an incremental cost of $100,000 per quality-adjusted life year gained, how do we determine whether this is cheap or expensive?
For all of these reasons, we may need to reach deeper into our evaluators' toolkits, using not only economic methods but also other ways of determining what VfM is.”
Farida: “That is a good point. Each analysis needs transparent criteria for the final evaluative judgement. And they need to be tailored to the case at hand. What is your advice on good ways to go about VfM assessment?”
Julian: “I think we need to think about what VfM means in a particular context. For example, Michael Scriven's Logic of Evaluation suggests we need to start by understanding the context and purpose of the evaluation, then we need to establish criteria of merit (the aspects of VfM that matter), performance standards (for example, what's the difference between good, poor or excellent VfM?), gather and analyse evidence of performance against the standards, and then synthesise the results into an overall judgement. Our overall judgement of VfM would need to explicitly address the question of whether the project outcomesare enough to justify the resources used.
Within this overall approach we might determine that efficiency is (or isn't) an important criterion of merit in a particular instance, and that given the characteristics of the evaluation subject, economic analysis is (or isn't) an appropriate way to address this. We might also identify a range of other criteria that could be addressed using other methods.
If our evaluation practice follows a deliberative democratic approach (such as that proposed by House and Howe 1999), we would seek to understand VfM from a range of perspectives and use appropriate processes and tools to make sense of it all. When we unpack what is of value to end users, we might find it looks very different to what is valued by funders.”
Farida: “What do you think are VfM pitfalls to avoid?”
Julian: “Oh, just the usual ones really, as with any evaluation. We need to avoid oversimplifying or overcomplicating. We need to avoid slavishly following methods of any type with insufficient thought about context and what really matters. We need to work within real world constraints to provide a quality evaluation for the time, money and data available while navigating the political environment. With regard to the latter we can expect to find ourselves in ideological debates about what VfM is and how it should be measured (e.g., efficiency versus equity considerations, quantitative versus qualitative measures, economics versus other evaluation methods). ”
This conversation isn't an end in itself. We are hoping to use this as a jumping off point to generate VfM learning scenarios. If you are interested in this topic, let us know about the issues you've faced in trying to apply the concept of VfM. And tell us about the practical responses you've developed. We'd like to gather as many stories as possible and use them to develop some interactive scenarios to help in developing appropriate VfM responses.
Resource
Evaluation methods for assessing Value for Money
This paper by Farida Fleming reviews different evaluation methods used for assessing Value for Money, compares their similarities and differences, and examines the questions each method is most suited to evaluating. The paper gives an overview of how to conduct Cost Effectiveness Analysis, Cost Utility Analysis, Cost Benefit Analysis, Social Return on Investment, Basic Efficiency Resource and ranking correlation of cost vs impact.