52 weeks of BetterEvaluation: Week 25: Evaluators have feelings too: Two sides of the evaluation coin
BetterEvaluation recently published a new paper, ‘Two sides of the evaluation coin,’ exploring what can happen when miscommunication, changing leadership and misunderstanding disrupt the smooth running of an evaluation: and what can be done to minimise these risks.
Authors from both the evaluator and commissioner side wrote the report jointly. John Rowley, who was part of the evaluation team, has blogged on the paper, saying that ‘it deals with issues that profoundly affect program evaluations but which are almost never shared in an open and public way.’ His fellow-evaluator, Pete Cranston, has also blogged about what the experience taught him about the role of evaluation in learning, and the role of failure. Now their co-author Penelope Beynon, who was a commissioner for the evaluation, shares her side of the story, and argues for the importance of recognising the emotions involved in a bumpy evaluation ride:
In 2011 I was involved as commissioner in an evaluation of a knowledge sharing network, AfricaAdapt. The evaluation itself was carried out by a team from CommsConsult with John Rowley and Pete Cranston. We all agree that despite best intentions on both at times the process was painful and emotional for all involved, and while it did eventually result in a useful report (and with us parting as friends) I think we all wish it could have been a gentler ride.
Our reflective account of the evaluation experience, was recently published by BetterEvaluation. Two years and 11,000 miles away from the evaluation experience, I was surprised at how strongly I felt when I re-read our reflection report this week. I remember feeling upset and frustrated at the time about misunderstandings and that we didn’t seem to be able to get on the same page despite all of our repeated efforts. In fact, at times it felt like the more we tried to resolve issues the less we understood each other - perhaps because we each became more entrenched in a defensive position. In his recent blog (archived link) on the topic, John Rowley also alludes to the emotions he felt. While we obviously would all prefer it when the emotions we feel during an evaluation are things like pride, excitement and satisfaction, negative emotions are important too: it will be a sad day when commissioners and evaluators do not care enough about the work that they do to feel a bit upset when things go wrong – to this end, I’ll take ‘upset’ over ‘indifferent’ when ‘elated’ isn’t on offer.
I also want to comment on some of the things that we did right:
Firstly, soon after the contract was complete we (commissioner and evaluators) participated in a joint After Action Review (AAR) to unpack and document our experience. At that stage in the process emotions were still fairly raw, and since none of us were under any obligation to reflect individually or together I think it is commendable that we did it. The AAR helped us to pinpoint the factors that gave rise to issues and to come up with a few solutions for next time.
Secondly, in our reflections we did not force a consensus. Through the After Action Review we identified different perspectives and different understandings about the evaluation process and its purpose – some of which persist today. Where possible we resolved these misunderstandings, and where not possible we acknowledged with respect that there were differences. You’ll see this expressed as two narratives in the report. Consensus sometimes masks a power struggle and I’m pleased that we allowed both of our views to co-exist rather than either never publishing our experience at all or one party forcing the other to accept their perspective as ‘truth’ – both of which looked like possible outcomes at times.
Finally, we swallowed our collective pride and shared this experience with others!
Resources
Unusually perhaps, this article is written jointly by the commissioner and the contractor of a recent evaluation. The process was rich, complex, at times difficult and a learning experience for all of us involved. Quite extraordinarily, we agreed to review the process of this evaluation between us and had already started sharing ideas on what could be learned from our experiences before finding this opportunity to contribute to BetterEvaluation. Cranston, P., Beynon, P. and Rowley, J.
- Reflecting, evaluating and triple loop learning
Euforic Services Blog - Pete Cranston
- Two sides of the evaluation coin (archived link)
Oxford Network of International Consultants Blog - John Rowley