Week 35: What is Visionary Evaluation?

kaleidoscope

Today we start a series on "visionary evaluation" - the theme of the 2014 American Evaluation Association conference in October.  The series is designed to encourage discussion of these issues to inform presentations and discussions at the conference and to allow a global conversation about them.

Thanks to the BetterEvaluation folks for this opportunity to discuss the AEA conference theme, Visionary Evaluation for a Sustainable, Equitable Future. The theme is intended to spur us to draw forth the best of what evaluation can offer to our world in these times of diminishing natural resources and ongoing social conflicts and inequities.

Visionary evaluation is not a particular method but rather is shorthand for encouraging evaluators to pay special attention to three kaleidoscopic twists they can put on their own particular work to support movement toward a sustainable, equitable future. Those twists are (a) attending to systems thinking, (b) building relationships, and (c) sustainable, equitable living. We’ll hear more about these topics in other blogs over the next few weeks.

When I first thought about this theme, I assumed the stakeholders of the initiative being evaluated would determine what was meant by a sustainable, equitable future. I didn’t think it was necessary for the evaluator to have his/her own vision of such a future. I’ve changed my mind about this: I now think that we all need to be conscious of our own vision of a sustainable, equitable future. Note: this is not about predicting the future or knowing how to get there. Rather it is envisioning what we each think it would look like when it existed.

Why the evaluators’ vision matters

As evaluators, evaluation managers, and evaluation users, we all bring our own biases to our work. Articulating these biases is important to help us understand that we are not totally objective in our work. I am realizing that my vision of a desired sustainable, equitable future is part of the bias I bring. I want to be as upfront as possible about what my orientation is and then be able to see the connection between it and that of the stakeholders with whom I am working. I’ve found that my vision of a desired future makes a difference in, for example, how I think about the boundaries of the evaluation. It shapes the conversations I have with the initiative leaders when starting an evaluation which in turn shapes the questions used in data collection and analysis.

One person I talked to recently argued that evaluators are to be neutral and having a certain view of a desired future would not represent neutrality. I tend to think in terms of evaluators being fair and transparent rather than neutral. Part of transparency is being transparent with ourselves, knowing ourselves so we can be aware of our own biases when interacting with the stakeholders of the initiative we are evaluating. What are your thoughts on this?

What is your vision of a sustainable, equitable future? Do you think it matters if you have such a vision? Why?

Perspectives from a few evaluators

Sue Soal, Work Process Consultant, Community Development Resource Association, South Africa

I have struggled to find concise responses to these questions, and have had to revert, as I so often do, to biography and context: 

I work sometimes as an evaluator, out of an OD background, and tend towards the narrative, to participation, reflection and iterative learning as a means of strategy development. What’s more, I work mainly with NGOs, social activists and community focused initiatives.
 
Perhaps then it’s no surprise that I think – ‘Isn’t “a sustainable, equitable future” itself a vision? … For sure, that’s my vision.’

These two words trigger a multitude of associations, all of which place me firmly in the camp of progressive, future-focused, inclusive, activist, transformatory, planet-minded people. Even realistic and sensible, I’d like to think. The details, we can squabble about amongst ourselves. And what’s more, the stakeholders of the initiatives I work with are in that camp too.
 
So as an evaluator,  I see myself as intensely partisan. I am on the side of the people I serve. And I am on the side of the side that they are on. 

I am fortunate that these are almost always my clients. That said, as an evaluator, the contribution I can make to strengthening that side is precisely to be ‘aware of my own biases’ and, crucially, to help them to be aware of their own. To help them to see themselves, as if from the outside. 

To ‘see’ might include recognition of both unspeakable failure (lack of impact in given outcomes) and shameful blind-spots (poor organisational conditions, including weak strategic ability). 

But that , I think, is the job of the evaluator: To help the client and wider stakeholder system  to see clearly, and to account for this frankly. To do this effectively, I need to be able to stand before them, to win their trust, and then, ever so carefully and thoughtfully, to come to stand alongside them - not face-to-face but shoulder-to-shoulder – so that together, we look at the same thing (evidence included) and, in that moment, with the same eyes.

Does that make me a visionary evaluator? I kind of hope so.

Farida Fleming, Research and Evaluation Consultant, Assai Consult, Australia

The identity crisis of evaluation: I often think we evaluators want to be something other than we are. Some evaluators feel bad that we are an auxiliary service. We sit alongside programmes. They do exciting things: deliver services, create change, transform relationships. We ‘just’ assess their progress. 

The trouble with wanting to be something else, is you don’t realise what you are. You’re too busy mimicking others, who already are good at what they do, that you end up do nothing well.

I agree with prioritising relationships and acting in a sustainable and equitable manner (I’m don’t know exactly what ‘attending to systems thinking’ means for my evaluative practice). But I don’t agree that this makes my evaluation ‘visionary’. These aren’t my biases. They are my values. And they help me make evaluative judgements. I need to be transparent about my values. And I need my stakeholders to articulate their values too. Because it may be that they want to evaluate a program based on other values than mine. 

So I don’t think we need ‘visionary evaluation’ to make a better world. Regular evaluation, with clearly articulated values is good enough. We just need to do it well.

Resources

For background and description of the theme, go to the AEA website. You’ll find a coffee break webinar, a short video, and a written description of the theme. You’ll also find links to the conference plenary sessions and two presidential strand sessions in which conference attendees will share their ways of thinking about and enacting the theme.

To learn more about conceptualizing a future that simultaneously attends to the economy, social issues, and natural resources (an orientation congruent with how I envision a sustainable, equitable future), see Bob Willard’s website. Bob will be a conference keynote speaker. Also check out The Natural Step especially their work on Future Fit Benchmarks.

'Week 35: What is Visionary Evaluation?' is referenced in: