What does it mean to ‘un-box’ evaluation?

a child's hand reaches into a box of large chalks

This guest blog by Jade Maloney is the first in a series about un-boxing evaluation – the theme of aes19 in Sydney, Australia.

The series is designed to generate a global discussion of the theme ‘un-boxing evaluation’ and what that means for our profession and practice. Jade Maloney is co-convenor of aes19. She is also a Partner at ARTD Consultants, specialising in design and evaluation with people with disability and in the disability sector.  

As evaluators, we see evaluation as a gift. We see evaluation’s potential to support effective policy and program design, guide ongoing program development, provide insight into on-the-ground practice, and identify whether intended (and any unintended) outcomes are being realised. We see how evaluation can support better public policy and, thereby, better individual, social and environmental outcomes. But we also know that not everyone sees evaluation this way.

Why do we need to un-box evaluation?

A lot has been written on the lack of use of evaluation, something that approaches such as Utilization-Focused Evaluation have been designed to counteract. My own research with Australian Evaluation Society members found that there are a range of barriers to evaluation use that evaluators encounter, including politics, timing, disinterest, resistance, fear of judgement and lack of clarity about purpose.    

These are things that I’ve come across in my own practice, in particular I’ve worked with people who fear of being judged throughout the evaluation process. A range of factors can underlie this fear – including fear of failure, conception of evaluation as an accountability exercise rather than a learning opportunity, and conflation of personal performance reviews with program evaluation.

We know that for some – perhaps, even many – evaluation is an imposition, an uncertain undertaking, or simply an unknown, with its seemingly incomprehensible insider jargon. It’s only when you connect on a human level – listen, seek to understand, and respond in context – that you can convince these people that evaluation at least won’t be a hindrance and might actually be helpful. If you do your job well, perhaps they’ll even see it as a gift. This is at the core of un-boxing evaluation.

How do I hope aes19 will help un-box evaluation?

For me, un-boxing evaluation is about opening evaluation up to end users to enable understanding and ownership (often through conversations and capacity building) and being open to what can be learned from community. It requires a conversation about the basis on which we value, who does the valuing and the boundaries of evaluation.

I work with people with disability and am very much guided by the philosophy of “nothing about us without us”. I’ve learned a lot from peer researchers and from people coming to the task of evaluation with a different perspective than my own. So I’m keen to hear from other evaluators using co-production to un-box evaluation.

As an evaluator with lived experience of mental illness who sometimes evaluates mental health policies, I’m also keen for aes19 to continue the conversation about evaluating with a lived experience that Joanna Farmer’s presentation at last year's AES conference addressed. When conceptions of evaluator as a neutral outsider dominate, how do we recognise people with lived experience as more than program beneficiaries, evaluation participants or translators? As Patton, channelling Bob Stake, says “evaluators do not have to pretend neutrality about the problems innovators are attacking in order to do fair, balanced and neutral evaluations of their programs”.

What does this mean for the conversations we’re having about pathways to advance professionalisation within the context of the Australasian Evaluation Society? Who should and will be able to call themselves an evaluator? Will you have to be an evaluator to do evaluation? What’s the line between doing evaluation and being involved in evaluation?

In empowerment evaluation communities are in control with evaluator as coach. Where is line in defining who is the evaluator in this case? And what can the evaluator-coach learn from the communities they work with?  We’ll be able to have this conversation with David Fetterman and ask him about how the communities he’s worked with have shaped his practice.

What does un-boxing evaluation mean to you?

Un-boxing evaluation is not a straightforward process. It raises a lot of questions about what it means to be an evaluator and what it means to do evaluation. We’re hoping that aes19 will advance these conversations and that you’ll be inspired to join in. The call for papers for aes19 is open until Thursday 7 March 2019 – we look forward to hearing from you! 


'What does it mean to ‘un-box’ evaluation?' is referenced in: