What is it?
Evaluation, which means to assess the value or worth of something, is essentially about values. Underpinning R,M&E systems are questions such as 'Is this good? Which is better? What is best?' Therefore, it is important to be systematic and transparent about the values that are used through the development of criteria and standards, and where these come from. Identifying what success looks like should also take into account outcomes and impacts (intended and unintended, especially possible negative outcomes), processes (in particular consistency with values about ethical behaviour and non-violence), and the distribution of costs and benefits (in particular the comparative value of initiatives that work for most people on average and those that are particularly effective for the most vulnerable or disadvantaged).
It can be helpful to work through the logic of evaluation systematically - identify what the criteria are for success (for example, reduced incidence of violence against children), what the standards are (for example, a 10% reduction from the previous year; or a reduction to the national average; or a reduction to zero), and how diverse evidence will be synthesised (how different elements will be combined). Being clear about synthesis is especially important when there is an overall evaluative judgement, such as value-for-money which takes into account both effectiveness and cost - at what point is a more expensive option better? It is also important when there is a 'hurdle' requirement which must be met - for example, a cheaper option would be not acceptable if it involved the use of child labour.
Developing an agreed statement of 'what success looks like' generally involves a combination of drawing on formal statements of values, articulating tacit (unstated but important) values, and negotiating between the relative importance and legitimacy of different values. Formal statements of values include stated goals and objectives, Sustainable Development Goals, agreed standards, evaluative criteria and benchmarks (where these exist already), and the OECD-DAC criteria of relevance, effectiveness, efficiency, impact, sustainability. Processes that can be used to articulate tacit values include: hierarchical Card Sorting (HCS): a participatory card sorting option designed to provide insight into how people categorise and rank different phenomena, Photovoice: using cameras to allow participants (often intended beneficiaries) to take and share photos in order to identify what is important to them, rich pictures: exploring, acknowledging and defining a situation through diagrams in order to create a preliminary mental model how it works (including what is valued), Stories of change (part of the Most Significant Change approach) showing what is valued through the use of specific narratives of events, values Clarification Interviews: interviewing key informants and intended beneficiaries to identify what they value, and Values clarification public opinion questionnaires: seeking feedback from large numbers of people about their priorities through the use of questionnaires. Negotiating between different values can be done through: Concept Mapping, Delphi Study: generating a consensus without face to face contact by soliciting opinions from individuals in an iterative process of answering questions, Dotmocracy: recording participants opinions by using sticky dots to either record agreement or disagreement with written statements, Public Consultations: conducting public meetings to provide an opportunity for the community to raise issues of concern and respond to options.
Information about all of these is available through site that includes comprehensive information about criteria and standards..
Determining what success looks like in C4D
Applying the C4D principles
UN Agencies like UNICEF often use the OCED-DAC criteria. While these are clear and reputable, they are also very broad and generic, and processes are needed to operationalise these for a particular initiative. The C4D Evaluation Framework would encourage the following approaches:
Whose values are being used as the basis of the evaluation? What do stakeholders and beneficiaries consider to be good, better, and best C4D processes, practices and outcomes? How can participatory techniques (such as hierarchical cards sorting) be employed to effectively engage with stakeholders about what they value, and why?
|Whose criteria and standards are reflected are whose are excluded? What are the assumptions? Could the vision of success be enriched through the inclusion of different perspectives?|
|An holistic approach to this task encourages us to think about how the context influences the definition of success, values, aspirations and perspectives. It can be useful to seek ways to define holistic visions of success, beyond indicators and targets (i.e. in Results Frameworks) which often only show a single dimension of success.|
|Working with community groups, partners and others to find agreement about what success might look like means that everybody knows and understands what values are used to make judgements about a program. In other words, the criteria and values to judge success are shared and transparent.|
Recommended options and adaptations for C4D
Hierarchical Card Sorting
Hierarchical Card Sorting (HCS) is a participatory sorting and ranking process which helps to articulate participants' tacit criteria, standards and approach to synthesis. It could be used pre-implementation to describe criteria and standards and weigh them up against each other (i.e. which ones are most important?). Alternatively, it could be used post data collection to weigh up the value of different cases based on emergent, tacit values and standards. This approach is consistent with the C4D Evaluation Framework in the following ways:
- participatory: Hierarchical Card Sorting enables a participatory approach to describing criteria and standards and applying and weighing up different values.
- holistic: Hierarchical Card Sorting is a way to develop criteria and standards that are relevant and responsive to the context, rather that starting from global and generic standards.
- accountable: Because Hierarchical Card Sorting is a way of eliciting values from different groups, it is a way of ensuring social and downward accountability (especially when used pre-implementation).
Most Significant Change
Most Significant Change can be used post-implementation and involves processes of comparing and ranking to ascertain which changes are seen as most valuable by key groups. The process involves collecting stories of change, analysing and sorting these into groups, and then ranking to decide on the most significant or valuable changes. It is consistent with the C4D Evaluation Framework in the following ways:
- participatory: the process involves working with groups of stakeholders to collect stories and analyse what different group's value and consider most important about a programs impacts.
- complex: The strength of Most Significant Change is the way is it sensitive to unpredictable and emergent impacts (mainly positive).
Caution: in general Most Significant Change will not be sufficient as an R,M&E plan on its own, since it is mainly useful for picking up positive impacts at the extreme (less common) end.