Week 35: What is Visionary Evaluation?

Beverly Parsons's picture 4th September 2014 by Beverly Parsons

Today we start a series on "visionary evaluation" - the theme of the 2014 American Evaluation Association conference in October.  The series is designed to encourage discussion of these issues to inform presentations and discussions at the conference and to allow a global conversation about them.

Thanks to the BetterEvaluation folks for this opportunity to discuss the AEA conference theme, Visionary Evaluation for a Sustainable, Equitable Future. The theme is intended to spur us to draw forth the best of what evaluation can offer to our world in these times of diminishing natural resources and ongoing social conflicts and inequities.

Visionary evaluation is not a particular method but rather is shorthand for encouraging evaluators to pay special attention to three kaleidoscopic twists they can put on their own particular work to support movement toward a sustainable, equitable future. Those twists are (a) attending to systems thinking, (b) building relationships, and (c) sustainable, equitable living. We’ll hear more about these topics in other blogs over the next few weeks.

When I first thought about this theme, I assumed the stakeholders of the initiative being evaluated would determine what was meant by a sustainable, equitable future. I didn’t think it was necessary for the evaluator to have his/her own vision of such a future. I’ve changed my mind about this: I now think that we all need to be conscious of our own vision of a sustainable, equitable future. Note: this is not about predicting the future or knowing how to get there. Rather it is envisioning what we each think it would look like when it existed.

Why the evaluators’ vision matters

As evaluators, evaluation managers, and evaluation users, we all bring our own biases to our work. Articulating these biases is important to help us understand that we are not totally objective in our work. I am realizing that my vision of a desired sustainable, equitable future is part of the bias I bring. I want to be as upfront as possible about what my orientation is and then be able to see the connection between it and that of the stakeholders with whom I am working. I’ve found that my vision of a desired future makes a difference in, for example, how I think about the boundaries of the evaluation. It shapes the conversations I have with the initiative leaders when starting an evaluation which in turn shapes the questions used in data collection and analysis.

One person I talked to recently argued that evaluators are to be neutral and having a certain view of a desired future would not represent neutrality. I tend to think in terms of evaluators being fair and transparent rather than neutral. Part of transparency is being transparent with ourselves, knowing ourselves so we can be aware of our own biases when interacting with the stakeholders of the initiative we are evaluating. What are your thoughts on this?

What is your vision of a sustainable, equitable future? Do you think it matters if you have such a vision? Why?

Perspectives from a few evaluators

Sue Soal, Work Process Consultant, Community Development Resource Association, South Africa

I have struggled to find concise responses to these questions, and have had to revert, as I so often do, to biography and context: 

I work sometimes as an evaluator, out of an OD background, and tend towards the narrative, to participation, reflection and iterative learning as a means of strategy development. What’s more, I work mainly with NGOs, social activists and community focused initiatives.
 
Perhaps then it’s no surprise that I think – ‘Isn’t “a sustainable, equitable future” itself a vision? … For sure, that’s my vision.’

These two words trigger a multitude of associations, all of which place me firmly in the camp of progressive, future-focused, inclusive, activist, transformatory, planet-minded people. Even realistic and sensible, I’d like to think. The details, we can squabble about amongst ourselves. And what’s more, the stakeholders of the initiatives I work with are in that camp too.
 
So as an evaluator,  I see myself as intensely partisan. I am on the side of the people I serve. And I am on the side of the side that they are on. 

I am fortunate that these are almost always my clients. That said, as an evaluator, the contribution I can make to strengthening that side is precisely to be ‘aware of my own biases’ and, crucially, to help them to be aware of their own. To help them to see themselves, as if from the outside. 

To ‘see’ might include recognition of both unspeakable failure (lack of impact in given outcomes) and shameful blind-spots (poor organisational conditions, including weak strategic ability). 

But that , I think, is the job of the evaluator: To help the client and wider stakeholder system  to see clearly, and to account for this frankly. To do this effectively, I need to be able to stand before them, to win their trust, and then, ever so carefully and thoughtfully, to come to stand alongside them - not face-to-face but shoulder-to-shoulder – so that together, we look at the same thing (evidence included) and, in that moment, with the same eyes.

Does that make me a visionary evaluator? I kind of hope so.

Farida Fleming, Research and Evaluation Consultant, Assai Consult, Australia

The identity crisis of evaluation: I often think we evaluators want to be something other than we are. Some evaluators feel bad that we are an auxiliary service. We sit alongside programmes. They do exciting things: deliver services, create change, transform relationships. We ‘just’ assess their progress. 

The trouble with wanting to be something else, is you don’t realise what you are. You’re too busy mimicking others, who already are good at what they do, that you end up do nothing well.

I agree with prioritising relationships and acting in a sustainable and equitable manner (I’m don’t know exactly what ‘attending to systems thinking’ means for my evaluative practice). But I don’t agree that this makes my evaluation ‘visionary’. These aren’t my biases. They are my values. And they help me make evaluative judgements. I need to be transparent about my values. And I need my stakeholders to articulate their values too. Because it may be that they want to evaluate a program based on other values than mine. 

So I don’t think we need ‘visionary evaluation’ to make a better world. Regular evaluation, with clearly articulated values is good enough. We just need to do it well.

Resources

For background and description of the theme, go to the AEA website. You’ll find a coffee break webinar, a short video, and a written description of the theme. You’ll also find links to the conference plenary sessions and two presidential strand sessions in which conference attendees will share their ways of thinking about and enacting the theme.

To learn more about conceptualizing a future that simultaneously attends to the economy, social issues, and natural resources (an orientation congruent with how I envision a sustainable, equitable future), see Bob Willard’s website. Bob will be a conference keynote speaker. Also check out The Natural Step especially their work on Future Fit Benchmarks.

A special thanks to this page's contributors
Author
United States of America.

Comments

veronica.olazabal@nuruinternational.org's picture
Veronica Olazabal

Thank you Beverly for starting this discussion. I work in the international development sector, and have had the experience of being on the funder, fundee and even client side. These many lenses have allowed me to view evaluation as the bridge to more effective program planning, service delivery, and poverty eradication, especially around complex issues like equity and sustainability. 

This particular discussion is quite timely as the international development sector is discussing the post-2015 development agenda that will incorporate sustainability and equity much more than it has in the past. In my view, one of the reasons we are at this monumental place in history is because of the work evaluators have contributed to documenting, measuring, questioning, and deconstructing the rationale and underpinnings of the previous development goals. 

Given we (as evaluators) are often tangentially part of these program and policy discussions, as Farida alludes to, I believe we are more part of these discussions than ever before. And similar to Sue, my overall motivation and vision for more effective solutions around economic development is situated within the evaluation sector's ability to use its theories, practices and tools to drive toward a more sustainable and equitable global state for all.

Beverly Parsons's picture
Beverly Parsons

Thanks to Sue, Farida, Bojan, and Veronica for the comments. I'd like to pick up on Veronica's comment about the post-2015 development agenda and the role of evaluation in regard to the previous development goals. The more evaluators understand multiple meanings, theories, and practices related to sustainability and equity, the more we will be able to contribute to meeting future development goals. What sources are you using to learn about sustainability and equity?

Patricia Rogers's picture
Patricia Rogers

It's interesting to consider which organizations include sustainability as part of evaluations routinely.  It's very common in evaluations in international development and is included as one of the criteria in the OECD-DAC criteria.  I wonder if some of these examples might be useful to evaluators who are not working in international development. You can find more information about the OECD-DAC criteria and examples of their application here  http://betterevaluation.org/evaluation-options/dac_criteria

veronica.olazabal@nuruinternational.org's picture
Veronica Olazabal

In addition to the great references already added, I would point to what the next set of goals may be called, which builds off of the global policy discussions held in Rio+20 a few years ago on sustainability. The Millennium Development Goals (MDGs), which defined a lot of what was measured by the international development community after 2000 will likely be renamed the Sustainability Development Goals (SDGs). To learn more I suggest this link on the United Nation's website. The UNDP also released a video a bit back that I think captures the spirit and vision of what the post-2015 agenda aspires to (speaking of vision), which has equity and sustainability front and centre.

​I can also point to the "Outcome Document" that emerged from the Rio+20 meetings that frames out the direction of what the post-2015 agenda will strive for. As we know from the previous 15 years, this agenda will guide decision-making around what gets prioritised, funded and therefore measured in the development space.

Finally and specific to our evaluation sector, leading thinkers such as Marco Segone (UN Women) and Ada Ocampo (UNICEF) speak about how to build an equity focused lens into evaluation planning and analysis across their work and offered free through resources such My M & E.

Hoping this provides some additional food for thought for anyone wishing to learn what will be prioritised in the international development space post-2015, visionary evaluation around equity and sustainability, and how the American Evaluation Association's conference this year is quite timely as we move into the International Year of Evaluation (2015)!

Cheers!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.