ACTION-LEARNING; ADAPTIVE; CAPACITY DEVELOPMENT; CRITICAL REFLECTION
In a learning-based approach RM&E is integrated into the whole programme cycle and involves all staff and stakeholders. This principle draws on some of the core principles of action learning and participatory action research (PAR), including iterative reflection on implementation for continual improvement. Involving a broad group of stakeholders in R,M&E requires attention to capacity development and learning processes and events.
Where do we start?
It can be useful to begin by Reviewing R,M&E (meta evaluation) with a focus on previous efforts in order to understand: what worked well? what didn't work well? who was involved? how can the current initiative build on that?
To consider how learning fits into a new R,M&E initiative begin by Deciding on the purpose, and more specifically how the primary intended users intended to use R,M&E can help to clarify the expectations in terms of learning from R,M&E, and how this balances with accountability-focused purposes.
Develop R,M&E capacity, which should begin with an assessment of existing capacities, can also be a starting point for implementing the learning-based principle. It is especially useful to think about how capacity building processes can support participatory approaches to R,M&E.
Incorporating and implementing learning-based approaches in practice
Manage (and commission) an evaluation or evaluation system
Understand and engage stakeholders: To effectively implement the C4D Evaluation Framework, a receptive organisational and community context and culture is required. Staff of organisations at all levels and relevant community members need to be willing to engage in constant reflection and learning from R,M&E in order to continually develop and improve organisational systems and C4D initiatives. This is dependent upon meaningful stakeholder engagement in the beginning and continuing throughout implementation.
Establish decision making processes: Decision making processes and structures (such as an ongoing technical working group) should emphasise leadership and responsibilities for knowledge management, exchange and utilisation to ensure continuous learning, mutual understanding and creative ideas and thinking.
Decide who will conduct the R,M&E: In contexts where it is difficult to find available, local evaluators with the skills and knowledge to be able to undertake C4D evaluation and studies, partnerships with capacity building components can be considered.
Develop Planning Documents (Evaluation Plans and MnE Frameworks): Learning events, structures and processes (inclusive of all partners and community groups involved in implementation) should be built into M&E Frameworks and Evaluation/Research Plans. M&E Frameworks should be flexible enough to accommodate emergent issues. Some organisations are starting to refer to 'Monitoring, Learning and Evaluation Frameworks' to emphasise the importance of considering how frameworks can support learning in addition to producing information.
Document management processes and agreements: Recruiting consultants with expertise in both C4D and the specific program area can be challenging. It is even more difficult when seeking local or regionally based consultants. Consider what kinds of expertise are required, what kinds are desirable, and what kinds are easily translatable from similar fields and approaches. Also consider whether capacity building and mentoring partnerships can be incorporated to fill gaps. See also Decide who will conduct the research/evaluation (or other study or monitoring).
Reviewing R,M&E (meta evaluation): Including review or meta-evaluation processes in C4D R,M&E systems is a key part of being learning-based, using critical reflection processes, and it contributes to capacity development. The aim is to continually strengthen and improve R,M&E processes so that they better meet the needs of the people and organisations involved and help to create more sustainable, learning-oriented C4D organisations and initiatives.
Develop R,M&E capacity: Including capacity development processes in C4D R,M&E systems is a key part of being learning-based. This process should begin with a preliminary assessment of R,M&E capacities of local groups and institutions. What sort of ongoing training, support or mentoring might be needed? What sorts of local research training institutions are available? How can this best be delivered?
Develop initial description: This process should be seen as open to revision as the R,M&E proceeds and new learnings emerge that have implications for the focus of the M&E.
Developing a program theory/logic model: Program Theories and logic models can be used at various stages of the program cycle. In a learning-based approach, these would be developed over time as more knowledge becomes available:
Deciding on the purpose: The approach advocated by the C4D Evaluation Framework is to use R,M&E processes for adaptive and learning-based process, so that findings can be fed into ongoing C4D activities. This is because most C4D activities are complicated or complex (to understand the nature of your activity see Complexity).
|Describe (to answer descriptive questions)||
Use measures, indicators or metrics: Indicator selection should be focused on the type of ‘summary’ information that can tell us whether or not the intervention is ‘on track’ in terms of its implementation and anticipated results. Where the intervention content or implementation needs to be very adaptive and/or the results cannot be fully defined in advance (such as in complex situations), different indicators may need to be selected at different times during the intervention period. The indicators should help to answer the ‘key learning questions’ that are posed at various times.
Data management: Related to the participatory approach, it is important to consider whether stakeholders may need capacity building support to be able to effectively manage data.
Understand causes (to answer questions about causes and contributions)
Investigate Causal Attribution and Contribution: The learning needs may determine which combination of strategies will be most useful. While designs creating a counterfactual (strategy 1) are best in situations where strong hypotheses (theories) are known and need to be tested and proven, they are not as well suited in more exploratory situations. A combination of Strategy 2: Check the results support causal attribution and Strategy 3: Investigate possible alternative explanations (strategy 3) can be used where there is a need to learn about and better understandings of causes and changes.
|Report and support use||
Supporting use: This task contributes to a learning-based approach through taking seriously the tasks associated with supporting the use of findings in future programs and phases.
Challenges and strategies
People often assume that 'learning' is just a given part of any R,M&E that has recommendations and lessons learned section of the report, and assume that no special attention or planning is required.
|Being learning-based is both an approach, and a set of deliberate processes and strategies. It's an approach that requires more flexible, adaptive systems to allow for initiatives to grow and change as more understanding is developed. Deliberate processes and strategies, such as Reviewing R,M&E (meta evaluation), Developing R,M&E capacity, and thorough processes to Generalise findings, Develop recommendations, and support use are key to a committed learning-based approach. Examples may include learning-committees, annual reviews etc.|
|People sometimes assume that being 'learning-based' demonstrates a lack of commitment to accountability.||The C4D Evaluation Framework would suggest that learning is a form of accountability. In most cases these two purposes can be balanced. See the Decide purpose task to work through the options.|
|How can we be adaptive and emergent in our approach using Results Frameworks?||It is quite hard to be adaptive and emergent when using Results Frameworks. Although they can be adjusted at certain times, they require heavy planning and are difficult to use for truly adaptive and emergent implementation. There are other options. See Develop Planning Documents (Evaluation Plans and M&E Frameworks) for details of options compatible with adaptive and emergent approaches.|
IDEAS Guide - an entry-level guide designing a learning-focused M&E Plan for C4D initiatives. The guide supports implementing teams to lead the design of the M&E plan. Click here to go directly to the guide. This guide is consistent with the C4D Evaluation Framework in the following ways:
- Learning-based The resource has a strong emphasis on using M&E during implementation to adjust the project direction as the project unfolds. It uses a metaphor of sailing to help users understand what a learning-based approach mean.
- Participatory Each module in the resource outlines a group-based, often visual activity to explore options and make decisions in participatory ways in order to be accessible for people with little or no prior experience of M&E.
- Realistic The resource was developed in the context of small-grants, so it is sensitive to the needs of small-scale initiatives. The language and processes are as simple as possible.
- Accountable The resource includes a number of steps to map stakeholders and understand who has an interest in the findings of the M&E about the project. This includes funders, and may include others such as governments, community leaders, participants and others.
Community Radio Continuous Improvement Toolkit - This toolkit is premised on a mix of self-assessment and peer-review towards co-learning and horizontal evaluation. In this case, it is fellow community radio station staff and volunteers who undertake the assessment. It was created in the context of community radios in India, but, with some adaptation of the questions, the processes and guidance could be applied to support peer-assessment between organisations doing a range of different types of C4D. Click here to read a summary about when and why you could use this resource here. This resource is consistent with the C4D Evaluation Framework in the following ways:
- Learning-based The peer-review methodology enables staff and volunteers and different community radio stations to learn from the experiences and innovations of other radio stations while undertaking the peer assessment. Often peer-reviewers come back to their own stations with new ideas to adapt. The purpose of the methodology is continual learning.
- Participatory The peer-review process is participatory, with peer community radio station staff and volunteers engaging in critical and self-reflection meetings and workshops.
- Realistic The process outlined is focused on working through and reflecting on a series of questions. Although it takes some time, it makes use of the knowledge and expertise of participants and is not overly burdensome.
It is important to consider the following:
- Accountable Although the Indian Government has agreed to use this methodology in place of accountability- focused evaluation processes, the methodology is not about replicating external accountability R,M&E and is instead explicitly directed towards co-learning and continual improvement, towards building a community of practice. The methods of self-assessment and peer-review may not be considered rigourous enough in other circumstances where upward accountability and reporting are included as a purpose of R,M&E efforts.
Equal Access Community Researcher manual - This manual is part of the Participatory Monitoring and Evaluation Toolkit developed with a C4D organisation (Equal Access). The Community Researcher Manual was developed for the community researchers working on a particular C4D project. It clearly explains the approach, the role of community researchers, the context and the tools to be used. It is consistent with the C4D Evaluation Framework in the following ways:
- Learning-based This is an example of a resource developed to build the capacity of community researchers
- Participatory Including community researchers and building their capacity to lead the research is a good example of how to build in participatory approaches.
Participatory Rural Communication Appraisal (PRCA)
This is a C4D resource developed in the context of C4D and rural development by FAO, with wide applicability to other program focus areas. This is an excellent resource that provides guidance on how to work with community groups and institutions in participatory and learning-based ways to ensure that they are involved in deciding what kind of evidence and success they would like to generate from development interventions. Click to here to read a summary.
International collaboration for capacity building as part of the KAP study of VAC in Tanzania
In 2014 UNICEF Tanzania Country Office in collaboration with Government commissioned University of Huddersfield – the Centre for Applied Childhood Studies in partnership with Mzumbe University in Tanzania, to undertake a study entitled: The Drivers of Violence Against Children in Tanzania, a participatory action research exploring knowledge, attitudes and socio-cultural practices that contribute to violence against children in Tanzania Mainland and Zanzibar. This is an example of a learning based approach in the following ways:
- As part of the study a mentoring relationship between UK researchers and local university researchers was set up. This initiative brought together researchers from the UK (University of Huddersfield) and researchers from a local university (Mzumbe University in Tanzania) in a mentoring relationship.
- In addition, 10 researchers were recruited through Mzumbe University and trained to undertake community action research, and were deployed to research sites to recruit and train community researchers and child peer-researchers.
- The methods used in this initiative were not simply extractive (ascertaining people’s views through qualitative research), but used participatory approaches to actively engage with stakeholders in iterative processes of learning. The 500 participants were supported to explore and critically reflect on the socio cultural factors that underpin violence against children, and through gaining a better understanding of the issues, explore possibilities for change.
This example demonstrates the need to build the capacity in order to use participatory methods. The C4D Evaluation Framework would also highlight the following areas for consideration:
- Realistic Those interested in replicating should be aware that the involvement of the UK researchers increased the budget, and this in turn impacted on the sustainability of the capacity after the initiative ended. The mentoring model could have been more sustainable and cost-effective if a regionally-based mentor institution was matched with a local institution.
Terms of Reference for the action research and evaluation of ‘She Can’ - ActionAid
This is an example of a TOR for a learning-based, adaptive and emergent evaluation. Although the term 'C4D' is not used in this TOR, the activities include campaigns, mobilisation, coalition building, and women groups and school clubs: all relevant to C4D. The approach to be used as outlined in this TOR is a theory-based evaluation using some action research. Click here to read a summary and review of this resource. The approach and the TOR are consistent with the C4D Evaluation Framework in the following ways:
- Complex The use of the phased process allows for an adaptive approach. The first phase includes limited data collection to inform monitoring and learning strategies, followed by a second phase with six monthly data collection and review activities, and a third and final phase that includes a theory-based evaluation to unpack change processes.
- Learning-based Building on the phased, adaptive, and learning-based process above where findings are built into the change theory and implementation over time, the users (specified on page 9) are the program staff and partners who will use the findings to improve implementation, the 'beneficiaries' who will use it to better understand effective strategies for change, and DFID who are interested from a policy point of view.
- Participatory This TOR is an example of how an external evaluator can work with program staff to undertake evaluation. The description on pages 5-6 show clearly the way the consultant is expected to work in partnership with program teams and other stakeholders, and the governance structures outlined on page 9 point to the inclusion of stakeholders and partners.
- Realistic The TOR directly addresses this by stating that the evaluation design must be proportionate to the scale and scope of the project, and should seek to minimise the burden on project and partner field staff in particular' (page 8). Further, although the consultancy will last approximately 3 years over four countries, the budget is relatively modest at $100,000, accounting for the fact that it is not a full-time consultancy.
- Critical The TOR states that the evaluation design must give 'due consideration to the involvement of project participants at all stages, and must seek to give primacy to the views and voices of people living in poverty, particularly women and girls'.