About our project on 'Evaluation practice in Aboriginal and Torres Strait Islander settings'

The story behind the Project

When the project was started with a grant from the Department of Prime Minister and Cabinet (DPM&C), Indigenous Affairs, in July 2018, the need for cultural sensitivity and input from Aboriginal and Torres Strait Islander people was emphasised.

A Steering Committee and Advisory Group consisting of Aboriginal and Torres Strait Islander people and non-Indigenous people working in the evaluation sector was established. A number of people on the Advisory Group highlighted the need for a way to ensure appropriate conduct when managing valuable Indigenous Cultural Intellectual Property (ICIP) on the BetterEvaluation website as curated or co-created resources. The BetterEvaluation core team was referred to Terri Janke and Co, an Indigenous legal firm specialising in ICIP, who developed a contributor agreement. This document brings together Western and Aboriginal and Torres Strait Islander protocols to protect the Intellectual Property (IP) and ICIP of contributors to the project.

With a second grant from DPM&C, a working group of six consultants with expertise in evaluation in Aboriginal and Torres Strait Islander settings was established. The project team included Belinda Gibb & Sharon Babyack (Indigenous Community Volunteers), Donna Stephens (Menzies School of Health Research), Carol Vale & Debbie Hoger (Murawin Consulting), Kate Kelleher (Kate Kelleher Consulting) supported by Greet Peersman (BetterEvaluation). The team agreed that it was important to put time and effort into communicating the approach they would be taking and to ensure systems were in place to protect ICIP. This was seen as an important step to building trust with contributors before engaging with them.

The team also agreed that, rather than asking for contributors to share their IP and ICIP (such as existing guidelines and tools related to evaluation), we would instead look at existing good practice evaluations and provide communities* the opportunity to evaluate the evaluations. All team members were passionate about ensuring that the voice of Aboriginal and Torres Strait Islander People was at the heart of the project. It was agreed that only good practice evaluations that have a community evaluation would be included on the website. 

The story behind the Contributor Agreement and the Code of Conduct

The team adapted the initial contributor agreement form to reflect the shift in direction towards highlighting the voice of communities about evaluations affecting them. The document was also revised based on the approach the Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS) takes with their contributor agreement. The team wanted to ensure it was user friendly and appropriate for the contexts they would be using it in. A description of the project and its purpose was added, who BetterEvaluation is, the team members that are working on the project as well as an explanation of the risks and benefits of being involved in the project.

As well as being a formal process to keep the team accountable for the code of ethics (see below), the contributor agreement form is also an important discussion tool before any interview with the evaluators or community members*. It opens up opportunities for contributors to ask questions and feel confident and informed before they agree to participate. 

To guide the team’s engagement with evaluators and communities* and how examples of good practice are shared on the website, a formal code of conduct was developed. It addresses 10 key principles and how the team puts these into practice and is in line with the Contributor Agreement.

Please contact us if you would like to see our Contributor Agreement.

The story behind the Ethical Protocol

The team identified the need for an ethical protocol for evaluators to better protect the ICIP, rights and protocols of Aboriginal and Torres Strait Islander people and communities*. We shared stories of where ethics had failed, the damage this has caused and the injustice of how data can be misused, misrepresented through a Western lens and stolen from the people it should serve to benefit.

We recognised that there were many existing ethical frameworks and guides we could draw on, including the work we had already done with Terri Janke and Co for our team’s code of conduct. After undertaking a literature review, we identified the need to tailor existing work to the evaluation sector we work in. We categorised the principles and protocols from other ethical frameworks and guides and discovered the same themes emerging, differing in the specific language used. We consolidated these themes and, drawing on our team’s yarns, put them into our own words.

We identified the need to capture the six key themes that had emerged into an image that simplified the complexity of the theme and also illustrated that all, not just some, of the principles need to be implemented (a circle image, consisting of six parts). We also acknowledged the many barriers to good ethics that we had encountered in our own practice and created an image to show how the ethical principles, as well as courage, integrity and cultural humility are required to overcome these barriers (the circle image surrounded by barriers).

Finally, we explored how these principles would be applied in an evaluation context. We used the BetterEvaluation Rainbow Framework, contextualising each principle at each stage of an evaluative activity: from initial management  through to reporting.

Our Ethical Protocol is, at this stage, based on desk-based research and yarning among our group of Aboriginal and non-Indigenous evaluators. Some of us are independent consultants, some are internal evaluators, some work in research and evaluation institutions. While our experience and backgrounds are diverse, and a number of different Aboriginal groups are represented in our working group, we acknowledge that we need to test the Ethical Protocol further with other Aboriginal and Torres Strait Islander people and communities.

We developed a participatory questionnaire based on the six themes of the Ethical Protocol. Evaluators and communities* are invited to have their evaluations assessed according to the principles included or to provide feedback on the Protocol itself.

* The term ‘community’ is used to refer to a geographically based community defined by specific boundaries, a community organisation, or a community defined by a shared interest. Within the context of this project, an evaluation may not include interactions with all members of a community but could also be with one or more representatives of a community.

Acknowledgements

We would like to acknowledge and thank Maria Stephens, an Arrabi/Binning woman who speaks the Iwaidja language. She generously provided her artwork for this page.