What does it mean to ‘un-box’ evaluation?

JadeMaloney's picture 30th January 2019 by JadeMaloney

This guest blog by Jade Maloney is the first in a series about un-boxing evaluation – the theme of aes19 in Sydney, Australia. The series is designed to generate a global discussion of the theme ‘un-boxing evaluation’ and what that means for our profession and practice. Jade Maloney is co-convenor of aes19. She is also a Partner at ARTD Consultants, specialising in design and evaluation with people with disability and in the disability sector.  

As evaluators, we see evaluation as a gift. We see evaluation’s potential to support effective policy and program design, guide ongoing program development, provide insight into on-the-ground practice, and identify whether intended (and any unintended) outcomes are being realised. We see how evaluation can support better public policy and, thereby, better individual, social and environmental outcomes. But we also know that not everyone sees evaluation this way.

Why do we need to un-box evaluation?

A lot has been written on the lack of use of evaluation, something that approaches such as Utilization-Focused Evaluation have been designed to counteract. My own research with Australian Evaluation Society members found that there are a range of barriers to evaluation use that evaluators encounter, including politics, timing, disinterest, resistance, fear of judgement and lack of clarity about purpose.    

These are things that I’ve come across in my own practice, in particular I’ve worked with people who fear of being judged throughout the evaluation process. A range of factors can underlie this fear – including fear of failure, conception of evaluation as an accountability exercise rather than a learning opportunity, and conflation of personal performance reviews with program evaluation.

We know that for some – perhaps, even many – evaluation is an imposition, an uncertain undertaking, or simply an unknown, with its seemingly incomprehensible insider jargon. It’s only when you connect on a human level – listen, seek to understand, and respond in context – that you can convince these people that evaluation at least won’t be a hindrance and might actually be helpful. If you do your job well, perhaps they’ll even see it as a gift. This is at the core of un-boxing evaluation.

How do I hope aes19 will help un-box evaluation?

For me, un-boxing evaluation is about opening evaluation up to end users to enable understanding and ownership (often through conversations and capacity building) and being open to what can be learned from community. It requires a conversation about the basis on which we value, who does the valuing and the boundaries of evaluation.

I work with people with disability and am very much guided by the philosophy of “nothing about us without us”. I’ve learned a lot from peer researchers and from people coming to the task of evaluation with a different perspective than my own. So I’m keen to hear from other evaluators using co-production to un-box evaluation.

As an evaluator with lived experience of mental illness who sometimes evaluates mental health policies, I’m also keen for aes19 to continue the conversation about evaluating with a lived experience that Joanna Farmer’s presentation at last year's AES conference addressed. When conceptions of evaluator as a neutral outsider dominate, how do we recognise people with lived experience as more than program beneficiaries, evaluation participants or translators? As Patton, channelling Bob Stake, says “evaluators do not have to pretend neutrality about the problems innovators are attacking in order to do fair, balanced and neutral evaluations of their programs”.

What does this mean for the conversations we’re having about pathways to advance professionalisation within the context of the Australasian Evaluation Society? Who should and will be able to call themselves an evaluator? Will you have to be an evaluator to do evaluation? What’s the line between doing evaluation and being involved in evaluation?

In empowerment evaluation communities are in control with evaluator as coach. Where is line in defining who is the evaluator in this case? And what can the evaluator-coach learn from the communities they work with?  We’ll be able to have this conversation with David Fetterman and ask him about how the communities he’s worked with have shaped his practice.

What does un-boxing evaluation mean to you?

Un-boxing evaluation is not a straightforward process. It raises a lot of questions about what it means to be an evaluator and what it means to do evaluation. We’re hoping that aes19 will advance these conversations and that you’ll be inspired to join in. The call for papers for aes19 is open until Thursday 7 March 2019 – we look forward to hearing from you! 

We'd love to know your thoughts on what un-boxing evaluation means for you in the comments below.

A special thanks to this page's contributors
Author
Partner, ARTD Consultants.
Sydney/ Melbourne, Australia.

Comments

Anonymous's picture
Cyriaque Meka-Mevoung

In addition to the proposed definition, un-boxing evaluation should also include the capacity of telling the story (whether a success or a failure) as is, to the interested stakeholders.

JadeMaloney's picture
Jade Maloney

That's a really good point. Being able to share the story could help to un-box evaluation in multiple ways - not only instrumental but in building a culture in which evaluation is widely valued.
I think there's a lot more to tease out in un-boxing evaluation than I have said here.

Alice Macfarlan's picture
Alice Macfarlan

Kind of like a literal "un-boxing" of evaluation reports that have been filed away and left to gather dust... 

I agree this is a really important area. There needs to be more effort to make sure that evaluations are used - thinking about audiences' needs and preferences and using innovative reporting methods to meet these is one way forward, as well as (or hand in hand with) working towards a strong evaluative culture.

Jindra Cekan's picture
Jindra Cekan, PhD

We really must look at our values and at the values our evaluations could support or threaten in the commissioning organizations. In ex-post project evaluations we have found our findings have sometimes led organizations to claim sustained success drawing on findings from one program while ignoring equally compelling sustainability findings in other country programs... or worse yet, trying to hide the results entirely rather than learning from them! If we do not "un-box" results and learn from them, we miss opportunities to do better and learn from the very participants we're tasking to sustain such programming. Thanks!

JadeMaloney's picture
Jade Maloney

Thanks for your comment Jindra.

Agree, it is really important that we think about how to un-box evaluation results. For me this means thinking about how we communicate evaluation findings - how we can frame them not only for the owners of this particular program/ policy, but for others working in related spaces who can learn from them. 

Anonymous's picture
Nibador

Nice post

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.