In this guest blog, Fran Demetriou (Lirata Consulting and volunteer M&E advisor for the Asylum Seeker Resource Centre’s Mentoring Program) shares her reflections from the recent Australasian Evaluation Society (AES)'s 2018 conference, held in Launceston - in particular, what are some of the lessons a young and emerging evaluator might take away from the event? This blog was originally posted on the AES blog, which was launched at the 2018 conference and shares reflections and thoughts from the Australasian evaluation community.
The theme of transformations resonated with me. I’m relatively new to evaluation and it’s been an intense journey over the last two years in learning about what evaluation is and how to go about it well. This conference (my first ever evaluation conference) was a pivotal point in that journey.
As an ‘emerging evaluator’, my first question was… ‘what does that mean?’ I participated in one of the emerging evaluators panels, where one of the facilitators, Eunice Sotelo, did some excellent miming of the concept (I can’t justify it with text, so you’ll have to ask her nicely to demonstrate it). An audience member in the session called us caterpillars, following on from butterfly references in Michael Quinn Patton’s inspiring opening plenary. I’m not sure we have a working definition of transformation yet, but I’ve got some good imagery.
This caterpillar came to the conference with a good grounding in evaluation, but with a lot more to understand, including where I was at and what I needed to do to develop.
Here’s what I’ve taken away from my first AES conference:
Community spirit and failing forwards
I was struck by the diversity of content in the sessions. There is so much to learn about and so much innovation underway to enable us to better address complex social problems. This felt overwhelming as a newcomer, but I was comforted to find a community of evaluators at the conference who wanted to share, collaborate and learn from one another.
It was great to have so many interactive sessions to enable those connections. As an emerging evaluator, I also appreciated the focus given at the conference to welcome us into the community, focus on our development, and provide platforms to hear our perspectives on opportunities to develop the sector.
The emphasis on learning from failure was valuable. One of my conference highlights was Matt Healey’s interactive session (Learning from failure: A Safe Space Session) where, under Chatham House rules, evaluators with various backgrounds, specialisations and levels of experience shared some of those facepalm moments. It was comforting to know others had made similar mistakes to me, but even more beneficial to learn from others’ mistakes to help mitigate them in my own practice.
I learned that, as we continue to transform our practice to tackle complex problems, there are going to be failures along the way – and that’s ok, so long as we recognise them, learn and adapt. I went along to the panel session Umbrellas and rain drops: Evaluating systems change lessons and insights from Tasmania and listened as a highly experienced team shared the challenges they have encountered implementing systems change through the Beacon Foundation in Tasmanian schools. For me, it helped surface the importance of having strong relationships with partners and funders who are willing to fail forwards with us.
We have power! Let’s share it, empower others and be ready to let go
The conference reiterated for me the power that we hold as evaluators. We have the power to influence who is included in evaluations, and how – and we need to push back to make sure those who are affected by decisions are involved meaningfully in the process.
Through some enlightening role play, the We are Women! We are Ready! Amplifying our voice through Participatory Action Research (Tracy McDiarmid and Alejandra Pineda from the International Women’s Development Agency) session helped me to reflect on the ever-present power dynamics between evaluation stakeholders, and how to critically assess and address this to ensure stakeholders are included.
I learned that power isn’t just about how you include stakeholders, but what you bring to each evaluation through your own identity, and the often unstated cultural values you hold. A challenge I will be taking back to my practice is to be more critically aware of my own identity and the impact it has on evaluations I work on.
These conversations and discussions were summed up for me in Sharon Gollan’s and Kathleen Stacey’s plenary with the galvanising question: “When will YOU challenge the power when it is denying inclusion?”
It’s all about values
Very much connected to power is whose values are heard and counted in an evaluation. I went to several sessions dedicated explicitly to values in evaluations. It was exciting to see both the development of theory and the sharing of practical tools for eliciting values in evaluations.
In their plenary, Sharon Gollan and Kathleen Stacey provided a reminder that the benchmark for doing evaluation has been defined by the dominant culture. This was a powerful insight for me – it seems obvious, but it’s something easily overlooked. The way we undertake evaluation has cultural values embedded deep within it, and we must take care to think about the suitability of our approaches especially with Indigenous communities.
Being able to elicit values in each stage of an evaluation is a separate challenge altogether from understanding they are important. It was great then to have several sessions focused on identifying different types of values, articulating values approaches, specifying where values fit into an evaluation (at the start, and then they permeate everything), and how to work with these values, especially in culturally appropriate ways.
We like food metaphors
And finally, we must be a hungry bunch, because the sessions were peppered with food references.
Some savoury metaphors included policy being described as spaghetti, with evaluation making it a bento box (Jen Thompson in Traps for young players: a panel session by new evaluators for new evaluators), and a key takeaways slide with a pizza image (Joanna Farmer in When an evaluator benefits: the challenges of managing values and power in evaluating with lived experience).
Pudding was offered up by Jenny Riley’s and Clare Davies’ appetisingly named Outcomes, Dashboards and Cupcakes and Matt Healey’s ignite session on evaluators as cake Just add water: The ingredients of an evaluator.
My favourite food reference, reflecting the importance of power and values, was from Lisa Warner, who was quoted by a panellist in Developmental evaluation in Indigenous contexts: transforming power relations at the interface of different knowledge systems: “If you’re not at the table, you’re on the menu”.
I don’t know about you, but I certainly feel well nourished!
I’ll be transforming my work to better address values, power and inclusion, and I look forward to the Emerging Evaluators Special Interest Group kicking off soon to continue learnings with and from others.
Thanks for a great first conference, and I look forward to seeing you in Sydney next year!
What do you think?
If you attended AES18, we'd love to hear your own reflections below. What stood out for you and what do you think you'll incorporate as part of your practice? For those who didn't attend, are these reflections resonating with what's coming out of other conferences and events? Let us know in the comments below!
Fran Demetriou works at Lirata Consulting as an Evaluator, and volunteers as an M&E advisor for the Asylum Seeker Resource Centre’s Mentoring Program. Get in touch with Fran via LinkedIn and Twitter (@Fran_Demetriou).
*Please note that the original version of this article incorrectly quotes a Developmental evaluation in Indigenous contexts: transforming power relations at the interface of different knowledge systems panellist for saying: “If you’re not at the table, you’re on the menu”. In fact, the panellist was quoting Lisa Warner who said this in her STEPS team presentation. The post has been updated to reflect this.