Two Sides of the Evaluation Coin

Unusually perhaps, this article is written jointly by the commissioner and the contractor of a recent evaluation. The process was rich, complex, at times difficult and a learning experience for all of us involved. Quite extraordinarily, we agreed to review the process of this evaluation between us and had already started sharing ideas on what could be learned from our experiences before finding this opportunity to contribute to BetterEvaluation.

Here we try to reflect honestly on our experience and learning about how we might have avoided difficulties and increased the value of the overall process and the final products. For the most part, our story is told as one narrative because we share a common understanding of the events and issues that arose. But we are not always in agreement. Where our perspectives diverge, the report makes space for two narratives to show the two sides of the same evaluation coin.

This article will be useful for anyone planning an evaluation or for contracting parties to suggest where misunderstanding can frequently arise and how to head it off in advance.

This example comes from the virtual writeshop process led by Irene Guijt for BetterEvaluation.

Source

Cranston, P., & Beynon, P., & Rowley, J. (2013) Two Sides of the Evaluation Coin: An honest narrative co-constructed by the commissioner and the contractor concerning one evaluation experience. BetterEvaluation

5
Average: 5 (2 votes)
Rate this Resource:

Comments

Ann's picture
Ann Larson
Rating: 
0

I enjoyed this honest account of an evaluation. What particularly rang true was the misunderstandings arising from an interim or preliminary report. As an evaluator who really respects fieldworkers and managers I value feedback at this stage to make sure my findings resonant and to help understand what lessons they draw out. This is usually an important input into forming recommendations. But several times I have sent documents out too early, often missing sections that I think are self-evident or parts that are not polished only to skew the discussions from something that could have been constructive to one that is accusatory. It takes a lot of effort and sometimes isn't possible to regain a good relationship. I have the balance right on a current project. Unusually it is an evaluation which is showing that the program does not work at all. I am doing the analysis in steps, sharing the results (its a quantitative project) with the senior manager after each bit of analysis and asking her what I should look at next. I am hoping by the end of the exercise she will not only be resigned to a high profile program not having the outcomes she thought but also figured out how to do better for the target group. I should get her to write her story of the evaluation.

FaridaFleming's picture
Farida Fleming
Rating: 
0

Thanks very much for posting this resource. Having the perspective of the client and the evaluators - particularly when there are differences - makes for an illuminating and useful read.

I was interested in one of the final thoughts from the paper: that there was a tension between the need for a rigorous evaluation and a positive inquiry for learning. There are a few implicit assumptions in the way the tension is expressed. One is that people cannot or do not learn from rigorous evaluation. Another is that people learn (preferably? mostly? only?) from positive inquiry.

We know that learning from failure is important - think of the seriousness that airline companies treat crashes. But we know that admitting to failure (at any level) is difficult and can threaten a person's self-concept. Facilitating a 'safe space' for discussing failure is important. This includes helping people develop a separation between their own self identity and their work.

I wonder if there is more to the distinction (from the client's perspective) about what they could and could not learn from? Particularly, I wonder whether the tension is more about prospective learning and backwards focused findings. What I mean here is that perhaps the team was more interested in looking at positive stories because they could try and do more of that (and do less of what got in the way of that) in the future. On the other hand, there is not much that the team can do with findings that the network was doing ok given its age and stage.

petecranston's picture
pete cranston
Rating: 
0

Thanks for the comments.I've done more reflecting on the reflections here http://bit.ly/17ELkw0
* Ann, your experience is so similar, and yes, it would be good to see her blog
* Farida, I think your question will be better answered by Penelope, as one of the commissioners. I agree with all you say about the difficulty of coming to terms with 'failure', but that is as much to do with the term, I think, which is somehow so complete, large, absolute. In reality we need a constant set of adaptations, adjustments, as information comes back about progress. But I think there is something else to do with a project mindset: in order to get funds, and then to motivate teams, and maintain momentum, people in a project - and I speak for myself as well - can get caught up in a cycle of self-referential positive feedback, believing what we want to believe about the project, and looking for evidence that supports that. It's not necessarily a dishonest process, rather to do with the dynamics of a time-limited project. It's hard to think of a context in which people are actually constantly adjusting to reality, challenging their own assumptions, and convincing everyone to change adjust theirs and change course.

Miek van Gaalen's picture
Anna Maria (Miek) van Gaalen
Rating: 
0

Thank you for posting and sharing this. For me as a learner or emerging evaluator this is very useful and practical. I was wondering however, in how far there were also cultural factors in play here at several times of the process. Communication is key when it comes to expressing expectations, establishing or guiding a process, interpreting what has been agreed on or not, and also the value of written word. In addition, different cultures deal with criticism in a variety of ways, or even can interpret differently what would be considered criticism, or constructive criticism or feedback. I would be curious to hear how that was dealt with? Was the evaluation team composed of nationals as well,and in how far were the meetings taking into account different aspects of intercultural communication? Thank you in advance, I realise this article was published a while ago, but hopefully somebody can still elaborate on this.

 

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.