C4D: Determine what 'success' looks like
What is it?
Evaluation, which means to assess the value or worth of something, is essentially about values. Underpinning R,M&E systems are questions such as 'Is this good? Which is better? What is best?'. Therefore, it is important to be systematic and transparent about the values that are used through the development of criteria and standards, and where these come from. Identifying what success looks like should also take into account outcomes and impacts (intended and unintended, especially possible negative outcomes), processes (in particular consistency with values about ethical behaviour and non-violence), and the distribution of costs and benefits (in particular the comparative value of initiatives that work for most people on average and those that are particularly effective for the most marginalised or disadvantaged).
It can be helpful to work through the logic of evaluation systematically - identify what the criteria are for success (for example, reduced incidence of violence against children), what the standards are (for example, a 10% reduction from the previous year; or a reduction to the national average; or a reduction to zero), and how diverse evidence will be synthesised (how different elements will be combined). Being clear about synthesis is especially important when there is an overall evaluative judgement, such as value-for-money which takes into account both effectiveness and cost - at what point is a more expensive method better? It is also important when there is a 'hurdle' requirement which must be met - for example, a cheaper method would be not acceptable if it involved the use of child labour.
General Information
Developing an agreed statement of 'what success looks like' generally involves a combination of drawing on formal statements of values, articulating tacit (unstated but important) values, and negotiating between the relative importance and legitimacy of different values.
Formal statements of values include:
- Sustainable Development Goals
- (where these exist already)
- The OECD DAC criteria of relevance, effectiveness, efficiency, impact, and sustainability.
Processes that can be used to articulate tacit values include:
- (HCS) a participatory card sorting method designed to provide insight into how people categorise and rank different phenomena
- Using cameras to allow participants (often intended beneficiaries) to take and share photos in order to identify what is important to them
- Exploring, acknowledging and defining a situation through diagrams in order to create a preliminary mental model how it works (including what is valued),
- (Part of the Most Significant Change approach) showing what is valued through the use of specific narratives of events
- Interviewing key informants and intended beneficiaries to identify what they value
- Seeking feedback from large numbers of people about their priorities through the use of questionnaires.
Negotiating between different values can be done through:
- Generating a consensus without face to face contact by soliciting opinions from individuals in an iterative process of answering questions
- Recording participants opinions by using sticky dots to either record agreement or disagreement with written statements
- Conducting public meetings to provide an opportunity for the community to raise issues of concern and respond to methods.
Information about all of these is available in the Rainbow Framework including comprehensive information about criteria and standards.
Applying the C4D principles
UN Agencies like UNICEF often use the OCED-DAC criteria. While these are clear and reputable, they are also very broad and generic, and processes are needed to operationalise these for a particular initiative. The C4D Evaluation Framework would encourage the following approaches:
Participatory
Whose values are being used as the basis of the evaluation? What do stakeholders and beneficiaries consider to be good, better, and best C4D processes, practices and outcomes? How can participatory techniques (such as hierarchical card sorting) be employed to effectively engage with stakeholders about what they value, and why?
Critical
Whose criteria and standards are reflected and whose are excluded? What are the assumptions? Could the vision of success be enriched through the inclusion of different perspectives?
Holistic
An holistic approach to this task encourages us to think about how the context influences the definition of success, values, aspirations and perspectives. It can be useful to seek ways to define holistic visions of success, beyond indicators and targets (i.e. in Results Frameworks) which often only show a single dimension of success.
Accountable
Working with community groups, partners and others to find agreement about what success might look like means that everybody knows and understands what values are used to make judgements about a program. In other words, the criteria and values to judge success are shared and transparent.
Recommended methods and adaptations for C4D
HCS is a participatory sorting and ranking process which helps to articulate participants' tacit criteria, standards and approach to synthesis. It could be used pre-implementation to describe criteria and standards and weigh them up against each other (i.e. which ones are most important?). Alternatively, it could be used post data collection to weigh up the value of different cases based on emergent, tacit values and standards. This approach is consistent with the C4D Evaluation Framework in the following ways:
- Participatory: Hierarchical Card Sorting enables a participatory approach to describing criteria and standards and applying and weighing up different values.
- Holistic: Hierarchical Card Sorting is a way to develop criteria and standards that are relevant and responsive to the context, rather than starting from global and generic standards.
- Accountable: Because Hierarchical Card Sorting is a way of eliciting values from different groups, it is a way of ensuring social and downward accountability (especially when used pre-implementation).
MSC can be used post-implementation and involves processes of comparing and ranking to ascertain which changes are seen as most valuable by key groups. The process involves collecting stories of change, analysing and sorting these into groups, and then ranking to decide on the most significant or valuable changes. It is consistent with the C4D Evaluation Framework in the following ways:
- Participatory: the process involves working with groups of stakeholders to collect stories and analyse what different groups value and consider most important about a program's impacts.
- Complex: The strength of Most Significant Change is the way is it sensitive to unpredictable and emergent impacts (mainly positive).
Caution: in general Most Significant Change will not be sufficient as an R,M&E plan on its own, since it is mainly useful for picking up positive impacts at the extreme (less common) end.
Expand to view all resources related to 'C4D: Determine what 'success' looks like'
'C4D: Determine what 'success' looks like' is referenced in:
Framework/Guide
- Communication for Development (C4D) :
- Communication for Development (C4D) :
- Communication for Development (C4D) :
- Communication for Development (C4D) :
- Communication for Development (C4D) :
- Communication for Development (C4D) :
- Communication for Development (C4D) :
- Communication for Development (C4D) :