52 weeks of BetterEvaluation: Week 35: Social return on investment in evaluation
In this week’s blog we interview Wouter Rijneveld, a consultant working on measurement and utilisation of results, mainly in international development.
He recently published a paper on the use of the Social Return on Investment approach in Malawi and we wanted to find out about his experience of using this less-reported approach. We were doubly interested when he told us that he was initially skeptical about SROI.
Your paper describes the use of social return on investment in an evaluation, what is that exactly?
Social Return on Investment (SROI) derives from economic calculations which express the ratio between financial returns and financial investments. The difference between standard return on investment and SROI is that SROI includes more than just financial returns and attempts to reveal the economic value of social and environmental outcomes in the same units as financial returns. The technical part of SROI is similar to Cost Benefit Analysis (CBA), but the approach comes with a number of principles which take it beyond CBA. These principles are 1) involve stakeholders, 2) understand what changes, 3) value things that matter, 4) only include what is material, 5) do not over-claim, 6) be transparent, 7) verify the result.
How did you get involved in using SROI?
My client, Red een Kind, wanted to test the use of SROI and asked me to do an evaluation using SROI with one of their projects in rural Malawi. I was skeptical at first. It seemed there were too many assumptions with SROI and I thought it was too one-dimensional. I believe there is a lot of value in really listening to people in participatory workshops and hearing their version of the story, as well as in using numbers that represent some parts of reality. But bringing the whole judgment on a project down to one single number? “Your intervention returns xx dollars for every dollar invested”. OK, but what does that really mean?
So why was SROI selected in this case?
The project was a Community-Based Childcare Centre in Mteperera, Nkhata Bay in Malawi, which works in around fifty villages on early childhood development, after school programs, safe motherhood and other community initiatives. The purpose of the evaluation was to find out what effects the interventions had on people in rural communities. The reason SROI was selected was that the client had made a commitment to experiment with different approaches to measuring results. These other approaches had to do justice to the integrated nature of the interventions and should be empowering for stakeholders, rather than just extractive. This led to the choice of using SROI through participatory workshops.
How did you implement SROI here and what other methods were used?
The stakeholders worked together to develop a theory of change for the childcare centres and associated interventions. They then went on to calculate the investments in the project. Then the changes for each of the stakeholders were discussed: what had changed, for how many people and how could this be observed? The value of each of these changes was discussed.
Calculating these monetary equivalents of changes for each of the stakeholders wasn’t easy, but local ingenuity, economic insights and choice modeling went a long way. Although we couldn't establish with any accuracy the contribution of other actors and influences, this was included rather than assume 100% attribution to the childcare project. Most calculations were done on flipcharts, partly by the groups themselves, but some were also included in an ‘impact map’ in a spreadsheet. The next day all findings were reviewed by a smaller group of stakeholders to verify the data that were used, the methods of valuation and the calculations. During a second visit, the findings were presented to a large group from these villages and the experience was discussed in a focus group discussion.
Having seen it in practice, how have your views on SROI changed?
One of the things that helped me come back less skeptical than I began was the SROI principle of choosing the less ambitious option for any and every choice (“do not over-claim”). For example, to determine the value of time, the stakeholders had heated debates on how much they would be able to make in a day, but we settled down on a much lesser rate of MK325 (the equivalent of just under a dollar at the time) which was used in 'cash for work' programmes.
It is very interesting to realize that SROI explicitly builds on the assumption that value is almost by definition subjective. In economic theory, prices are the result of subjective valuations of products which create a demand. Applying SROI through participatory workshops can have an empowering effect by legitimizing subjective valuations by stakeholders. In our case one of the most interesting findings was the calculation of the value of investments: after calculating the monetary equivalents of their own voluntary investments and contributions, community members realized that their own investments were over 80% of the total costs. This led to the village headmen to publicly praise all volunteers involved.
I haven’t become a full blown believer in SROI. It is still just an option among others for me. But it is an option that deserves a legitimate place in the whole range of approaches to monitoring and evaluation. Many aspects of SROI are not particular to this approach, for example, developing a theory of change, the principle of involving all relevant stakeholders, finding the changes that specific stakeholders experienced, or thinking about relative contribution. But the unique aspect of SROI is where changes and inputs are all expressed in the same monetary unit to enable comparison. All principles surrounding this essence, including a very active community of practitioners, a system of trainings and accreditations and a growing body of good practices and examples, serves to guard against overly simplistic applications and increases the credibility of the approach.
What tips would you give other evaluators or commissioners of evaluation considering using SROI?
It is important to keep in mind that SROI does not provide methods to measure changes, but a framework to value and compare changes within a set of given principles. If there is a need for robust measurement of changes additional approaches will be needed. The same applies to the question of contribution : SROI does acknowledge this, but does not provide the methods to determine the relative contribution of the intervention to each of the changes. It is possible to apply additional approaches to determine contribution. SROI could best be used when lots of data are available (e.g. from research in similar situations) or when it is acceptable to work with stakeholder perceptions of change. SROI is meant to provide insight and judgment on the value of change rather than very precise data.
When using SROI it is important to know and apply the principles, e.g. to involve stakeholders instead of making it a desk exercise. Developing an audit trail where all calculations can be traced is also important in order to avoid figures and ratios coming as a surprise from a black box. It is also very helpful to connect to the very active community of SROI practitioners, where there is a lot of exchange of methods of valuation and interpretations of the principles in practice.
What does the investment return?: Read the full report on the use of SROI to evaluate these childcare interventions by Red een Kind in Malawi.
The SROI network: The Social Return On Investment (SROI) Network promotes the use and development of the Social Return on Investment methodology internationally, encouraging a community of practice along the way.+
[Image: A community workshop during the Community-Based Childcare Centre evaluation, Wouter Rijneveld]