Evaluating with the Littlewell Working Group

2019

Resources

Evaluating with the Littlewell Working Group

File type

PDF

File size

718.4 KB

In 2014, Indigenous Community Volunteers (ICV) embarked on a Monitoring, Evaluation and Learning (MEL) Review to build internal MEL capacity.

ICV’s approach to monitoring and evaluation

The monitoring and evaluation (M&E) undertaken with the Littlewell Working Group in 2014-15 was one of the first examples for Indigenous Community Volunteers (ICV) of applying the participatory approach and new monitoring system to co-design short and long term indicators of change with communities, monitor them throughout the project cycle and assess effectiveness at the end.

In 2014-15, ICV embarked on two projects with the Working Group:

(1) Littlewell Submission Writing Project (2014-2017); and,

(2) Littlewell Oral History Project (2015-2018).

The participatory M&E approach was built into the design and delivery of these community-led development projects that the Working Group asked ICV to assist with.

Self-determination is central to all ICV community development and evaluation projects. In practice this means that ICV only works at the invitation of communities and all projects and M&E work are led and owned by Aboriginal and/or Torres Strait Islander people and communities.

The Littlewell Working Group invited ICV to work with them. This process was formalised through an explanation of how the relationship would work and signing a project agreement.
The Working Group described their priorities. These were used to co-design the project objectives and project plan.
As part of the project plan, short term (project) indicators and long-term (dream) indicators were co-designed with the Working Group.

Extensive time and energy was invested at the start of the project to understand the Working Group’s priorities, key relationships, history and cultural practices and protocols. Observation and listening were key to this process. In the words of the ICV evaluator:

“So, monitoring begins when we first engage with the community… For me, to effectively measure change we need to hear the voice of the people and to see and feel the ripples of change. Hearing the voice of the people captures a powerful picture that empowers Aboriginal and Torres Strait Islander communities to have ownership of their future to create a better and brighter tomorrow…

When a community sets their own goals, they have a much better chance of success. Most communities already know the solution to the challenges they face, all we need to do is empower and support the community to drive and own the space and the process.”
— Doyen Radcliffe, ICV Regional Manager

A combination of methods was used to collect data relating to the indicators of change: observation, appreciative inquiry, transect walking, ten seeds technique and the use of images, video and voice recorders assisted with monitoring the activities and capturing multiple perspectives and tracking the journey.

As part of ICV’s MEL Review, importance was placed on the collection of a baseline. ICV’s first encounter with the Littlewell Working Group was in 2012. The ICV evaluator describes the baseline this way:

“In 2012, I met with the former residents of Littlewell Reserve who requested assistance from ICV. Thomas and a former resident who I will refer to as Brother Boy spoke about this pram to me and its significance to the story of Littlewell Reserve. This pram belonged to Brother Boy and as a toddler he was taken away and he lost contact with his family and the Littlewell community. Brother Boy was part of the stolen generation. Years later, Brother Boy began his journey to find his roots back at Littlewell. The pram may be only a pram but it symbolises the journey of the Littlewell peoples’ experience and where they want to see Littlewell Reserve in the future – it is the foundation of their vision – it is the baseline to moving forward and a strategy to how they will do this.”
— Doyen Radcliffe, ICV Regional Manager

Two men on a transect walk, looking at an old pram

Thomas Cameron, the elected spokesperson for the Littlewell Working Group, talking with ICV volunteer Peter White in 2015. Both men are looking at an old pram on Littlewell Reserve and Thomas is explaining the significance of the old pram.

How did the approach benefit the community? How were the findings applied?

The ICV evaluator explained that the participatory monitoring approach added richness in understanding the story of the community and was useful to deal with challenges and issues affecting the project.

Participatory monitoring supported governance and decision making by clarifying multiple perspectives and was used by the Working Group to refocus or change direction as required.

The participatory approach provided flexibility and “breathing space” to track where and why some things were moving slowly, while others had traction and moved more quickly. It provided the Working Group the opportunity to own the decision-making process and solve the challenges the projects faced.

“Flexibility is the key when working with communities as the community needs to own the process, the space and vision. Rigid structures do not work as things can change.”
— Doyen Radcliffe, ICV Regional Manager

The use of technology such as voice recorders, cameras and video camera were an effective way to capture the voice of the Working Group, word for word. This avoided the need to interpret what people were saying and it was less distracting and intrusive than the evaluator taking notes while people were talking.

A structured approach that included the Working Group’s identified short and long-term indicators provided guidance on where to focus efforts, and which information to collect and record during monitoring points and throughout the activity.

Establishing the baseline was key to effectively assessing change later on as described by the community and hearing the voice of the people. It provided a clear point of comparison.

Images enhanced the participatory monitoring experience when combined with words and/or audio.

The Ten Seed technique deepened the evaluators understanding of the significance of the Littlewell project. While it was anticipated that recording the stories of former residents would be positive, questions asked in the Ten Seeds technique revealed that for some participants, recording their stories was also a difficult experience as it prompted them to revisit memories of being taken away as children. However, while it was difficult, the participants also highlighted that being involved in the project and sharing their story was healing and represented a significant step for them personally.

“It's like my sister mentioned in one of her interviews, it's a place of healing. It's a place where we've been exposed to a lot of violence, and quite a few of our family's been taken away and put into missions. But, we survived, and the reason I say we survived... They knocked us down. We've been knocked to the ground, but we got up, and we continue on fighting for something that's really close to our heart.”
— Thomas Cameron, Littlewell Working Group elected spokesperson

The participatory monitoring process enhanced the leadership, governance and decision making of the Working Group. It enabled the Working Group to clarify direction at each monitoring point. The Working Group established that they were about halfway to achieving their aims at the end of 2015. In 2015, the Working Group identified that one of the aims had been achieved: “Use a video camera to record the stories of the people who lived at Littlewell so these memories are preserved as an oral history”. This monitoring point helped the Working Group to focus on taking the next steps to achieve their aims.

Key to the project was recognising the diversity of the Working Group that were no longer geographically connected. By identifying and working with the correct community member for the project, years of listening, observation and understanding were built into the project, drawing on the expertise of the elected spokesperson for the group who held this knowledge and co-designed the evaluation and project with ICV:

“I'm the sort of person, I like sitting down and listening to the old people yarning. I grew up knowing that, you know? And they've got an encyclopedia of knowledge. If we can get somebody to sit down and listen to them, they don't write it in books. They tell you the stories of the history of our people, and I guess that's rewarding to do that.”
— Thomas Cameron, Littlewell Working Group elected spokesperson

How were the ethical principles put into practice?

Referring back to the Ethical Protocol, we describe how the M&E work with the Littlewell Working Group put the principles into practice.

Prioritise self-determination, community agency and self-governance

Principle: Empowerment

Aboriginal and Torres Strait Islander peoples have the right to self-determination and to be encouraged and empowered in decision-making processes. Evaluators must listen and advise to the benefit of communities above all else.

This project worked with ICV’s ‘ground up’ community development cycle (that incorporates M&E): strong relationships, engagement and shared understanding came first, before any M&E activities started.

ICV was invited to work with the Littlewell Working Group after ICV spent time explaining how they would work with them, the benefits of the projects and how M&E would be used throughout.

The Working Group co-designed the project and M&E activities by prioritising the objectives and indicators for the project.

ICV undertook a transect walk with the Working Group to collect the baseline. ICV was invited by the Working Group to walk on Country and listen and learn about the significance of the site and the project the Working Group had planned. This was the Working Group’s chosen way of communicating with ICV. It was relational and, as it was chosen by the Working Group, gave the group power and agency to direct the evaluation and the project.

The Working Group participated in participatory monitoring sessions using the Ten Seed Technique where they collectively determined what had been achieved and what the future direction for the Working Group would be.

Capability strengthening that was meaningful to, and requested by, the Working Group was built into the project: partnership brokering, videography, interviewing and grant writing skills.

Principle: Diversity

Recognise the diversity and uniqueness of First Nations Cultures, Peoples and Individuals.

Observation was used extensively at the outset to slow down and take the time to understand the sensitivities of the project and the hardship and trauma that the Working Group had experienced.

Understanding the diversity of the Working Group and their history was key to this project, particularly as members of the group were no longer geographically connected together. ICV invested time to establish that their key contact did have the support of the wider group. The elected spokesperson for the group had invested much time and energy into travelling to visit group members and ensure the views and aspirations of the group members were reflected:

“I'm always saying ‘we’, but there has to be a leader somewhere to put this infrastructure into place, in a common sense way, and I was elected to do this so, in my view I didn't give up on the Mingenew mob, and living out of a suitcase, but the commitment made towards a vision, a dream, and listening to our elders gave me that vision and dream… the infrastructure in the beginning, the foundation, is to talk and consult with members before we go to government representation or to local government which was Mingenew Shire, at the time. And through consulting with, first and foremost, talking with our elders, and bringing it to their attention that we want to start working on Littlewell and try to preserve it. So, I would say, talk with the local community, and the local community are the key players in this respect…” – Thomas Cameron, elected spokesperson for the Working Group

ICV also informally confirmed that they were working with the elected spokesperson for the group through observations and interactions with group members.

Prioritise self-determination, community agency and self-governance
Principle	How the principle was put into practice
Empowerment	Aboriginal and Torres Strait Islander peoples have the right to self-determination and to be encouraged and empowered in decision-making processes. Evaluators must listen and advise to the benefit of communities above all else.	This project worked with ICV’s ‘ground up’ community development cycle (that incorporates M&E): strong relationships, engagement and shared understanding came first, before any M&E activities started. ICV was invited to work with the Littlewell Working Group after ICV spent time explaining how they would work with them, the benefits of the projects and how M&E would be used throughout. The Working Group co-designed the project and M&E activities by prioritising the objectives and indicators for the project. ICV undertook a transect walk with the Working Group to collect the baseline. ICV was invited by the Working Group to walk on Country and listen and learn about the significance of the site and the project the Working Group had planned. This was the Working Group’s chosen way of communicating with ICV. It was relational and, as it was chosen by the Working Group, gave the group power and agency to direct the evaluation and the project. The Working Group participated in participatory monitoring sessions using the Ten Seed Technique where they collectively determined what had been achieved and what the future direction for the Working Group would be. Capability strengthening that was meaningful to, and requested by, the Working Group was built into the project: partnership brokering, videography, interviewing and grant writing skills.
Diversity	*Recognise the diversity and uniqueness of First Nations Cultures, Peoples and Individuals.*	Observation was used extensively at the outset to slow down and take the time to understand the sensitivities of the project and the hardship and trauma that the Working Group had experienced. Understanding the diversity of the Working Group and their history was key to this project, particularly as members of the group were no longer geographically connected together. ICV invested time to establish that their key contact did have the support of the wider group. The elected spokesperson for the group had invested much time and energy into travelling to visit group members and ensure the views and aspirations of the group members were reflected: “I'm always saying ‘we’, but there has to be a leader somewhere to put this infrastructure into place, in a common sense way, and I was elected to do this so, in my view I didn't give up on the Mingenew mob, and living out of a suitcase, but the commitment made towards a vision, a dream, and listening to our elders gave me that vision and dream… the infrastructure in the beginning, the foundation, is to talk and consult with members before we go to government representation or to local government which was Mingenew Shire, at the time. And through consulting with, first and foremost, talking with our elders, and bringing it to their attention that we want to start working on Littlewell and try to preserve it. So, I would say, talk with the local community, and the local community are the key players in this respect…” – Thomas Cameron, elected spokesperson for the Working Group ICV also informally confirmed that they were working with the elected spokesperson for the group through observations and interactions with group members.
Inclusion	*Involve Aboriginal and or Torres Strait Islander people in all levels of the evaluation, from design phase right through to analysis and communicating findings.*	Initially, transect walking was chosen by the Working Group as their preferred way to plan and design the evaluation and project. This was an inclusive approach and gave the Working Group power and agency to influence and shape the project and evaluation. Video stories were also chosen by the Working Group. Again, as this was an approach chosen by the group, it was used to invite and welcome all members of the group to share their story. The Ten Seed Technique was suggested by the ICV evaluator. It was used at monitoring points to facilitate broad group ownership and governance and to clarify direction for future aspects of the project.
Facilitate control and data sovereignty
Community ownership	*Evaluation, and the knowledge created through evaluation, is owned by the community.*	The Working Group were most interested in owning their video stories. A YouTube channel was set up with the group as the administrators so they could have full ownership of the stories. The raw footage was also provided to the group. The story of the project and associated data is retained in ICV’s database. ICV is a custodian of this data, and the Working Group knows they can request to have it at any time. This data was not the key interest of the group. The stories generated were taken back to the Working Group for approval of use. The group was encouraged to use this information for their own purposes, such as grant applications or award nominations.
Respect	Respect the custodianship of Aboriginal and Torres Strait Islander peoples and cultures, their ongoing and unbroken connection to their land and water and the right to protect, maintain, control and benefit from their Aboriginal and or Torres Strait Islander Cultural Intellectual Property (ICIP).	As part of ICV’s interactions with the Working Group, stories of culture, Country, heritage and hardship and trauma were shared. ICV did not record or retain these stories in their database. These stories and related data were left with the community.
Interpretation of culture	*Aboriginal and Torres Strait Islander peoples have the right to determine and approve the interpretation of their culture as the primary guardians and interpreters of their cultural heritage.*	ICV took great care to develop a relationship with the Working Group whereby the group knew they could approach ICV at any time with concerns they may have about the management of their ICIP. ICV asked the Working Group before sharing any cultural stories that belonged to the group.
Integrity and authenticity of culture and knowledge	*Aboriginal and Torres Strait Islander peoples have the right to maintain the integrity of their ICIP.*	Before starting the project, ICV explained how monitoring would be incorporated into the project and that it would be an opportunity for the Working Group to direct the project, including the use of information collected.
Secrecy and confidentiality	*Aboriginal and Torres Strait Islander peoples have the right to keep secret their sacred and ritual knowledge in accordance with their customary laws.*	ICV chose not to share sacred stories and knowledge and instead recognised that these were the Working Group’s stories to share, in whatever manner they chose to do so. Within this project, the group shared some of those stories in video via a YouTube channel. The video stories were also intended to be linked to the heritage trail on their land.
Communicate transparently, build trust and obtain individual and community consent
Consent	*Obtain the free prior informed consent of Aboriginal and or Torres Strait Islander people before commencing the evaluation.*	Before starting the project, a project agreement outlining the project and M&E approach was discussed and signed. Importantly, the Working Group invited ICV to work with them. An ICV Community Development Officer (CDO) and ICV Regional Manager (RM) acted as key points of contact throughout the M&E activities and the Working Group knew they could raise any concerns with the CDO or RM at any time during or after the assessments. Thomas, the spokesperson for the Working Group, was asked permission before any information was recorded for M&E purposes.
Equity	*Evaluation must be transparent, equitable and respect the integrity of the community.*	The Working Group was an equal partner in the design of the M&E approach and the project. Ample time was built into the timelines for engagement and discussion.
Strengths-based recognition of cultures, acknowledging communities and individuals
Attribution	*Aboriginal and Torres Strait Islander peoples and communities have the right to choose to be acknowledged and attributed for their contributions to an evaluation.*	All proposed publication of work related to the M&E activities has and will be shared with the Working Group for comment and approval and to confirm appropriate attribution. This is an important benefit for the Working Group – recognition and celebration of their achievements along the way.
Strengths-based	*Affirm and celebrate culture. Take a strengths-based approach and build from cultural strengths.*	ICV worked to a strengths-based model whereby the conversations with the Working Group were directed towards the strengths of the group, the assets they had available to them and the progress they had already made, before ICV was even on the scene. For example, the group had already successfully forged a partnership with the local shire and obtained permission to preserve and use the land. The participatory approach throughout the project provided the opportunity for the Working Group to take the lead on framing the language for the project by discussing their vision for the future.
Strengthening of culture	*Aboriginal and Torres Strait Islander cultures are not static and Aboriginal and Torres Strait Islander peoples have the right to protect, maintain, revitalise and advance their cultures.*	The evaluator and volunteers undertook ICV’s cultural induction and worked closely with the elected spokesperson for the Working Group to ensure they were working to cultural protocols and had a clear picture of how the group wanted to advance their culture. To achieve this, an upfront investment in engagement and relationship building was allowed for. This allowed ICV to support the priorities of the group in tangible ways in areas that the group had identified and requested themselves. As a result, two projects were commenced that would benefit the community directly.
Participation	*Aboriginal and Torres Strait Islander peoples are evaluators and should be regarded as equal partners.*	By building M&E into the delivery of the projects, the Working Group took a key role in providing expertise on their culture, Country, ambitions and design of the projects. Formal monitoring points at start and end of the projects were useful points of reflection. In addition, ongoing discussion throughout project delivery was key. In particular, understanding the big picture and long-term vision of the Working Group and continuously reflecting on progress to achieve these goals. This was a key focus of the participatory monitoring where the Ten Seed Technique was used to determine how well the group’s indicators had been advanced.
Share benefits and apply two-way learning
Community transformation	*Aboriginal and Torres Strait Islander peoples involved in, or affected by, evaluation should benefit from the evaluation project and not be disadvantaged by it.*	The nature of ICV’s approach is inherently about community transformation as the community chooses projects that they believe will transform their community. This was also the case for the Littlewell Working Group. In addition, M&E findings were provided to the Working Group in written format. This information was drawn on by them for obtaining grants, for partnership proposals and after 2015, also for award nominations. The reflective M&E discussions throughout the projects informed the direction and next steps for the Working Group, and lessons were applied as the project progressed rather than waiting for a detailed write up at the end.
Community priorities	*Evaluation must reflect the priorities of the community.*	All assessments were based on community priorities. Short term (project) indicators and long-term (dream) indicators were co-developed.
Strengthen capacity	*Evaluation must build capacity and capability for decision making and voluntary actions of participants and the communities in which they live.*	Capacity-strengthening was built into the M&E approach, specifically relating to skills relevant to the community. Taking a participatory approach allowed for the delivery of tangible products (a YouTube channel with video stories, and a memorandum of understanding (MOU) with the local shire) while developing skills to use these beyond the involvement of ICV.
Share results	*Evaluation results must be presented and available to communities in a form that is translatable to community needs.*	At the outset, the Working Group advised what information would benefit the group and how they wanted it presented. The use of video recordings allowed for the translation of results into a format that benefited the Working Group: a YouTube channel with stories in the words of the group, edited in the style they chose. The group owns the channel.
Formalise accountability processes on ethical practice
Accountability	*Include appropriate mechanisms and procedures for reporting on ethical aspects of the research and complying with this ethical framework.*	Formalising the relationship and M&E approach was done through a project agreement before the project commenced. As ICV’s M&E processes evolved, the group was kept informed, including through the development of a brochure on how ICV undertakes M&E.

A special thanks to this page’s contributors

Thomas Cameron, elected spokesperson for the Littlewell Working Group; Yamatji Naaguja Wajarri man, Doyen Radcliffe, in his role as an internal evaluator and Regional Manager, ICV; Belinda Gibbs and Sharon Babyack, ICV.

We would like to acknowledge and thank Maria Stephens, an Arrabi/Binning woman who speaks the Iwaidja language. She generously provided her artwork for this page.

This is part of a series

These are good practice examples of evaluation and participatory monitoring with Littlewell Working Group. These examples are part of a bigger project which seeks to share examples of good evaluative practice from community across Australia.