This blog is the fourth in our series about un-boxing evaluation – the theme of aes19 in Sydney, Australia. The series is designed to generate a global discussion of the theme ‘un-boxing evaluation’ and what that means for our profession and practice. Catch up on the first, second and third blogs in this series here. This post's guest author Alicia McCoy (Head of Research and Evaluation at Beyond Blue) discusses her own experiences in 'un-boxing' evaluation by challenging her assumptions about what evaluation 'is', and how the NGO sector can benefit from this un-boxing.
When I first heard the AES conference theme was going to be on un-boxing evaluation, I was excited. I remember thinking that it is the perfect time for evaluation to have this conversation. As I have become more experienced in evaluation, I have become more aware of the ‘boxes’ it has been put in - by others and by myself.
Like Margaret Moon writes of her own experiences in her blog ‘Un-boxing the expert label’, I was not an evaluator when I started in an evaluation role (you can read more about my story in the journal article Pathways to becoming an internal evaluator: Perspectives from the Australian non-government sector) – like so many others, my passage into the world of evaluation was rather unexpected. Throughout my career – from hospital social worker to internal evaluator in a community-based organisation to leading research and evaluation in a national health promotion organisation – I have grown increasingly aware of the complexity of the world we live in. I appreciate more each day that there are systems operating within systems, and that these influence each other and make putting things into a box problematic. As much as we humans find comfort in making sense of the world around us by categorising things, doing this can raise issues because life isn’t easy or neat. We need to challenge the thinking that tells us that this approach is always the best one. This also means critically reflecting on the box we have placed evaluation in to work out what’s next for evaluation and where it can offer the greatest value.
Many years ago, at the beginning of my evaluation career, I remember sticking to what I thought were relatively straightforward facts and fundamentals of evaluation. Evaluation was this, it will do this and this and that, and it will give us the answers we need – we just need to trust in the process. With good intentions, I had put evaluation in a box - probably in an attempt to help me make sense of it. But like many evaluators, with time and experience, I learned nuance and discretion, and this altered my frame of reference and ultimately how I practised as an NGO evaluator. I now believe more than ever that we live in a world of partial knowledge. While evaluation can help reduce the uncertainty in the world around us, and inform the programmes and activities we design in an attempt to make things better, it doesn’t have all the answers – it simply cannot. Despite this, it remains a critical undertaking for any NGO.
It is an interesting and exciting time to work in the NGO sector. There has been a surge in recognition that many of the problems the sector seeks to solve or mitigate are wicked and complex. This has led to a wonderful rise in social innovation that I believe holds great promise for the sector. Organisations are understanding that they need to work together in new and different ways if they are to move the needle on the complex issues we face in modern society. Doing this well will take time and evaluation can support this journey. To a degree, one might consider these changes as a sort of un-boxing of social programming. What is exciting for NGO evaluators is how we work on un-boxing evaluation in concert with this.
As we do this un-boxing as evaluators, it is critical that we reconcile our discovery with the box that our non-evaluator colleagues have placed evaluation in. Understanding and responding to this is what will truly contribute to stronger social programming. To do this well, we need to meet colleagues where they are at. I know from my PhD research that evaluation means different things to different people. For instance, there is still a belief, amongst many, that the purpose of evaluation is primarily one of accountability and it has been tucked firmly away in this box. Accountability to beneficiaries and funders is important of course, but I do believe that viewing evaluation solely through this lens can raise issues for organisations and create missed opportunities. Busting myths and shifting the focus of evaluation to be more about learning and ongoing improvement is, I think, key to un-boxing evaluation meaningfully in NGOs. Creating an enabling environment for this to occur gives organisations the best chance of wading through the complexity and uncertainty they operate in.
The questions that come to mind for me as we navigate this conversation include:
What are the fundamental contents of the evaluation ‘box’ that we should hold onto, lest we risk losing the essence of evaluation? What contents are we willing to be flexible about?
What related activities do we need to put into the box to be effective and best support social programming, which may be experiencing some un-boxing of its own?
As we work on answering these questions, one of the greatest values NGO evaluators can bring to bear in their working relationships is evaluative thinking. Evaluators can focus on starting conversations, supporting the ‘so what’ and ‘what’s next’, noticing signals for further exploration, asking curious questions, and fostering learning throughout the program cycle. Coming from a place of inclusivity, this approach ensures that the NGO evaluator not only remains relevant but also thrives, offers optimal value to their organisation, and has a key role in contributing to better outcomes for beneficiaries. I’m looking forward to unpacking these boxes further with colleagues working from diverse evaluation roles at the conference.
What does un-boxing evaluation mean to you?
Un-boxing evaluation is not a straightforward process. It raises a lot of questions about what it means to be an evaluator and what it means to do evaluation. We’re hoping that aes19 will advance these conversations and that you’ll be inspired to join in. Early-bird registrations close on Friday, July 5th – hopefully we'll see you there!
We'd love to know your thoughts on what un-boxing evaluation means for you in the comments below.