C4D: Critical

Power; difference; equity; multiple perspectives

Including different perspectives highlights the critical importance of paying attention to power. Our approach to R,M&E needs to actively address issues of equity and diversity by paying attention to gender, caste, class, ethnicity, age, status, education and other relevant differences. Design and implementation of RM&E can build upon the strengths and limitations of different evaluation approaches and methods; to find the right approaches for your evaluation questions and include all relevant voices and perspectives.

Where do we start?

It is important to pay attention to issues of unequal power and maintain a critical-thinking mindset while undertaking all RM&E tasks. This is vital for a meaningful participatory R,M&E approach, and the best defense against tokenistic participation and bias in RM&E. A good place to start is by developing a plan for Reviewing the RM&E (meta evaluation) to embed critical reflection throughout the RM&E planning and implementation process.

Incorporating and implementing participatory approaches in practice

Manage (and commission) an evaluation or evaluation system

Understand and engage stakeholders:

Ensure an equity lens when thinking about stakeholders. Make sure you are not just working with the easy-to-reach groups. Think about differences in voice and power within each stakeholder groups. While the inclusion of representatives can be a good way to ensure integration of marginalised voices it can also be problematic. Are representatives truly representative or are there differences in power and class within the group they represent? Is there a risk of wealth-bias, literacy-bias, roadside-bias and other biases identified by Robert Chambers?

Establish decision making processes:

It is important to critically reflect on and remove any barriers to participation in decision making (e.g. geography of meeting locations, frequency of the meetings, logistics, language, etc.).

Decide who will conduct the R,M&E:

What are the assumptions about who should conduct the R,M&E? What alternatives are there, and how might they be more or less inclusive of diverse voices? What kinds of qualities are important for a facilitator/ evaluator? How will might different facilitators influence power dynamics?

Define ethical and quality standards for R,M&E:

It is important to question existing sets of standards and their relevance in the local setting. We need to ask: whose interests and expectations are reflected in the quality and ethical standards? what are the assumptions embedded in the standards? what other perspectives are missing from those standards?

Develop Planning Documents (Evaluation Plans and MnE Frameworks):

It is important to reflect on power imbalances in the development of these strategic documents. Who has control over the creation and any adaptations to documents? How accessible are documents? Some types of strategic documents, such as Logical Frameworks, reflect Western styles of thinking and planning.

Review evaluation (meta evaluation):

Critical reflection throughout all aspects of the RM&E helps to maintain the quality of the RM&E and identify areas for improvement or extra attention. It is particularly important where participatory RM&E approaches are used in order to maintain an eye to issues of power and voice. Developing meta evaluation processes helps to formalise the processes and procedures that will incorporate this in to an implementable plan for regular critical reflection.

Develop RM&E capacity:

Lack of local capacity can lead to exclusion of local voices and perspectives. Partnerships and capacity building within local community groups and institutions is important so that so that there is genuine inclusion and contribution of local voices and perspectives. Pay critical attention to power dynamics in capacity building partnerships.

Define

Developing a program theory/logic model:

Program theories should consider how a program might work for different groups, particularly vulnerable and marginalised groups. Theories and models should be developed with and alongside groups that experience marginalisation. This helps to develop a program theory/logic model that is sensitive to what might work (and what doesn't) for whom in what circumstances.

Identifying potential unintended results:

Unintended results may not affect everybody, and adverse outcomes for minority groups may not be obvious in the data. A critical approach to the identification of unintended results (with contributions from local groups) is important for understanding how C4D initiatives are affecting the least powerful.

Frame

Identify primary intended users:

It is important to bring a critical lens to this process, and ensure that the primary intended users are not only those with formal, hierarchical power. The processes for engaging with primary intended users should address issues of power and control to ensure the needs and values of the less powerful are not excluded.

Specify the key R,M&E questions:

In C4D it is important that questions are framed in such a way that allows for multiple and diverse voices to contribute answers. This is important for descriptive questions, causal questions and evaluative questions.

Determine what 'success' looks like:

Whose criteria and standards are reflected are whose are excluded? What are the assumptions? Could the vision of success be enriched through the inclusion of different perspectives?

Describe (to answer descriptive questions)

Sample:

Sampling should pay attention to equity dimensions, and ensure that the most vulnerable groups are represented, and that the data is able to be disaggregated. Additional effort might be needed to get adequate coverage of more remote, more disadvantaged groups due to known biases such as: roadside bias, seasonal bias, pro-literacy bias, etc.

Use measures, indicators or metrics:

Indicators should specify the required data disaggregations (often this needs to include age, sex, income, levels of vulnerability etc.). Local groups and institutions should be meaningfully involved in the process of developing and using indicators. This inclusion of local perspectives and attention to equity reduces the risk of indicators incentivising easier reach to populations to achieve targets.

Collect and/or retrieve data:

Consider weaknesses of methods in terms of equity, power and voice. Critically reflect on how certain methods may distort, exclude or silence particular perspectives.

Analysing data:

The data analysis process should involve looking for differences, exceptions, and a critical analysis of power. Participatory data analysis processes can help draw out these differences. In these cases it is important to reflect on who should be involved in analysis, how to ensure meaningful contribution.

Visualising the data:

From a communication of results perspective, data visualisation can help as many groups as possible to engage with data and findings and make reports accessible.

Understand causes (to answer questions about causes and contributions)

Investigate causal attribution and contribution:

It is important to pay attention to the different ways that C4D initiatives affect different groups.

Counterfactual-based designs (strategy 1) can show differences experienced by different groups through data disaggregations (looking at different variables). However, mechanisms to create a comparison groups (such as incentives) may disguise power differences.

Critical reflection on power dynamics and inclusion might therefore make strategy 2: Check the results support causal attribution and Strategy 3: Investigate possible alternative explanations better options.

Synthesise

Synthesise data from a single study/evaluation:

Consider whose voices are included and excluded from the process of weighing up findings and making judgements.

Synthesise data across evaluations:

Ensure a critical and equity-focused approach by exploring what works for whom and in what circumstances.

Generalise Findings:

Consider who the initiative has worked for and where (who has it not worked for) and how this might this translate to other contexts (places, people and groups). When using participatory approaches to generalising findings, consider whose perspectives are included and silenced in this process.

Report and support use

Identify reporting requirements:

Critically reflect on the assumptions relating to reporting. It is important to ask: are there good reasons why reporting must take certain forms? are there ways in which certain reporting requirements exclude or favour certain groups? whose needs are being served by the reporting requirements?

Ensure accessibility:

Because of the nature of C4D, there is likely to be a greater emphasis on communicating with diverse groups. How might differences in age, status, gender, geography, as well as disability, literacy, language, and education affect access, both physical access and access based on abilities?

Developing recommendations:

There is a need to ensure that the recommendations includes a range of voices and perspectives, taking into account the power inequities between stakeholders.

Challenges and strategies

Who is excluded?

It can be difficult, especially when using participatory approaches, to know exactly what to do when certain people or groups are being excluded or silenced by more powerful groups. There is no easy answer to this, but recognising that power is present in participatory settings is a good first step. Solutions will vary from case to case, and creativity and group reflections will be important. For example, could you separate larger groups into smaller groups of similar people (e.g. groups of women, men, girls, community leaders, farmers etc.). A highly recommended resource to develop thinking about power and how it is present in your R,M&E practice is a discussion about different dimensions and aspects of power on the Participatory Methods website by IDS.