Most significant change

The Most Significant Change (MSC) approach involves generating and analysing personal accounts of change and deciding which is the most significant – and why.

Overview

MSC is not just about collecting and reporting stories but about having processes to learn from them - in particular, to learn about the similarities and differences in what different groups and individuals value.

Using MSC involves:

1. Deciding the types of stories that should be collected (stories about what - for example, about practice change or health outcomes or empowerment)
2. Collecting the stories and determining which stories are the most significant
3. Sharing the stories and discussing values with stakeholders and contributors so that learning happens about what is valued.

It provides some information about impacts (both intended and unintended) but is primarily about clarifying the values held by different stakeholders. While MSC is insufficient for impact evaluation by itself, it can be an important component of an impact evaluation in terms of identifying and communicating types of experiences that can be triangulated and validated using other methods.

MSC can be used for ongoing monitoring and for evaluation purposes. MSC is often used in cycles that can vary from fortnightly to annually. A common frequency is quarterly to coincide with quarterly reporting. In emergency contexts monitoring cycles may be more frequent. Decisions need to be made about the most appropriate reporting period, balancing the costs and benefits involved, and taking into account the reporting gaps of any existing M&E systems. More frequent cycles build familiarity with the MSC process but increases the cost in terms of participants’ time. Frequent reporting will also increase the cost of the process, in terms of the amount of participants’ time taken up by the process.

Steps in the process

There are ten steps in the full MSC process:

1. Start and raise interest: Introducing MSC and fostering interest and commitment; this may involve specific training and identifying MSC champions.
2. Define the domains of change: Selected stakeholders identify broad domains of change, such as changes in people’s lives. A domain such as lessons learnt can capture negative changes.
3. Define the reporting period: Decide how frequently changes will be monitored.
4. Collect significant change stories: Ask participants and field staff simple questions such as, ‘Looking back over the last month, what do you think was the most significant change in [particular domain of change]?’
5. Select the most significant stories: The selection of stories uses a hierarchical structure. Stories collected by field staff are discussed by level one selection groups who decide on the most significant story in each domain. The selected stories are then considered by a smaller number of level two selection groups who decide on the most significant story. This process is continued a final selection group selects a single story for each domain. The number of selection groups will depend on the scale of the program and often on the number of layers of management that already exist. It is usually pragmatic for the level of selection groups to be based on existing organisational structures, however, new structures can be created to include broader perspectives. Story selection at each level involves: everybody reading the stories; in-depth discussion about which stories should be chosen; group decision on which stories are felt to be most significant; and, documenting reasons for the group’s choices. There is scope to be innovative in this method. Your project may not have a hierarchical structure so there may be other ways of forming groups around which the stories can be discussed and the values identified. The criteria for selecting stories can emerge and be documented during the discussion of reported changes. Alternatively, the criteria for story selection can be predetermined; however, determining the criteria before the stories have been read and discussed may limit openness to new experiences.
6. Feedback the results of the selection process: The criteria used to select stories are recorded and fed back to lower levels.
7. Verify stories: Selected stories are verified through site visits to check the accuracy of reported stories, and more detailed information about significant events is collected.
8. Quantification: This can occur in two ways, by including quantitative information in the initial story and by quantifying whether significant changes raised in stories at one location have also been included in stories at other locations.
9. Secondary analysis and meta-monitoring: Analysis of the MSC process such as who participated, how they affected content and the frequency of different types of reported changes.
10. Revise the system: Revise the design of the MSC process to take into account what has been learned from its use and analysis.

Causal Pathways features

How this approach might be used to incorporate features of a causal pathways perspective

A causal pathways perspective on evaluation focuses on understanding how, why, and under what conditions change happens or has happened. It is used to understand the interconnected chains of causal links that lead to a range of outcomes and impacts. These causal pathways are likely to involve multiple actors, contributing factors, events and actions, not only the activities associated with the program, project or policy being evaluated or its stated objectives.

MSC can be used in ways that incorporate the following features of a causal pathways perspective:

• Valuing actors’ narratives: When a causal pathways lens is added to an MSC approach, stories of change explicitly seek information about how people who are directly affected by a program understand how changes have (or have not) come about and how they value those changes.
• Addressing power and inclusion: MSC enables rather than directs participants who are encouraged to use their own judgement in identifying and selecting stories. MSC can be adapted to include participants at all levels of story selection. People who are closest to the events being monitored (participants and field staff) identify the stories that are relevant to them.
• Articulating explicit causal pathways: When a causal pathways lens is added to an MSC approach, it can help uncover how change has come about from storytellers’ perspectives.
• Understanding contextual variation: MSC utilises detailed descriptions of events within their local context and why they are significant.
• Paying attention to a range of outcomes and impacts: MSC involves making sense of events after they have happened, including unexpected and indirect results.
• Using an iterative, bricolage approach to evaluation design: The secondary analysis step in the MSC process uses data and learning to inform subsequent rounds of data collection and story selection.
• Taking a complexity-appropriate approach to evaluation quality and rigour: MSC includes a verification step that ensures the accuracy of selected stories. There is transparency about why particular stories are selected as most significant.

Background

History of this approach

MSC was initially developed by Rick Davies in 1996 to monitor changes in a large, complex development aid project with diverse implementation and outcomes as part of his PhD on organisational learning in non-government organisations. Jess Dart completed her PhD on MSC in 2000 and promoted its use in Australia. MSC has been used and adapted by a wide range of organisations in multiple countries and diverse settings.

Methods that are part of Most Significant Change

BetterEvaluation defines an approach as a systematic package of methods. The Rainbow Framework organises methods in terms of more than 30 tasks involved in planning, managing and conducting an evaluation. Some of the methods used in MSC and the evaluation tasks they relate to are:

• Develop the evaluation design
  • Joint evaluation design involves stakeholders in making decisions about the evaluation, including what to collect data on and how to interpret it.
• Establish decision-making processes: Deciding on the most significant stories may involve a range of decision-making processes.
  • Majority decision-making: Involves making decisions based on the support of the majority of the decision-makers.
  • Iterative voting: After a first vote people discuss why they voted as they did, this is followed by a second and possibly third vote hopefully moving towards consensus. If consensus is not reached a minority view can be documented, or there may be a split and equal support for two stories.
  • Scoring: Each person allocates each story a score and the story with the highest aggregate score is chosen.
  • Secret ballot: Each person votes anonymously followed by a discussion of reasons for the vote
• Collect and/or retrieve data
  • Personal stories: Narratives from individuals about their personal experiences
  • Purposeful sampling: Seeking stories on particular issues or from particular groups or individuals

Example

"In 1994 Rick Davies was faced with the job of assessing the impact of an aid project on 16,500 people in the Rajshahi zone of western of Bangladesh (6). The idea of getting everyone to agree on a set of indicators was quickly dismissed as there was just too much diversity and conflicting views. Instead Rick devised an evaluation method which relied on people retelling their stories of significant change they had witnessed as a result of the project. Furthermore, the storytellers explained why they thought their story was significant.

If Rick had left it there the project would have had a nice collection of stories but the key stakeholders’ appreciation for the impact the project would have been minimal. Rick needed to engage the stakeholders, primarily the region’s decision-makers and the ultimate project funders, in a process that would help them see (and maybe even feel) the change. His solution was to get groups of people at different levels of the project’s hierarchy to select the stories which they thought was most significant and explain why they made that selection.

Each of the 4 project offices collected a number of stories and were asked to submit one story in each of the four areas of interest to the head office in Dhaka. The Dhaka head office staff then selected one story from the 16 submitted. The selected stories and reasons for selection were communicated back to the level below and the original storytellers. Over time the stakeholders began to understand the impact they were having and the project’s beneficiaries began to understand what the stakeholders believed was important. People were learning from each other. The approach, called Most Significant Change, systematically developed an intuitive understanding of the project’s impact that could be communicated in conjunction with the hard facts.

Rick’s method was highly successful: participation in the project increased; the assumptions and world views surfaced, helping in one case resolve an intra-family conflict over contraceptive use; the stories were extensively used in publications, educational material and videos; and, the positive changes where identified and reinforced."

Example taken from Anecdote.com: Evaluating the soft stuff

Advice for choosing MSC

What types of projects and programs would this approach be appropriate for?

The ‘Most Significant Change’ technique – A guide to its use advises that MSC is likely to be valuable for programs that are:

• Complex and produce diverse and emergent outcomes,
• Large, with numerous organisational layers,
• Focused on social change,
• Participatory in nature,
• Designed with repeated contact between field staff and participants,
• Struggling with conventional monitoring systems,
• Offering highly customised services to a small number of beneficiaries (such as family counselling).

What types of evaluations is MSC appropriate for?

MSC is particularly useful when you need different stakeholders to understand the different values other stakeholders have regarding "what success looks like" - criteria and standards for outcomes, processes and the distribution of costs and benefits.

MSC can be very helpful in explaining HOW change comes about (processes and causal mechanisms) and WHEN (in what situations and contexts). Therefore, it can be useful to support the development of programme theory (theory of change, logic models).

What level and type of resources are required for this approach?

MSC is time-consuming; field staff are involved in collecting stories, and staff at each level of the organisation or program are involved in selecting which stories are considered most significant and why.

While specific training is not required, good facilitation skills and the ability to identify priorities are important. The ‘Most Significant Change’ technique – A guide to its use lists some of the key enablers for MSC:

• An organisational culture where it is acceptable to discuss things that go wrong as well as success
• Champions (i.e. people who can promote the use of MSC) with good facilitation skills
• A willingness to try something different
• Time to run several cycles of the approach
• Infrastructure to enable regular feedback of the results to stakeholders
• Commitment by senior managers.

How might this approach be part of an effective overall evaluation design?

MSC works best in combination with other methods for gathering, analysing and reporting data. It doesn't provide comprehensive information about the impacts produced by an intervention. It gives some information about impact and unintended impact but is primarily about clarifying the values held by different stakeholders. By itself, it is not sufficient for impact evaluation as it does not provide information about the usual experience but about the extremes. The ‘Most Significant Change’ technique – A guide to its use suggests that MSC could be complemented by evaluation approaches that provide:

• Quantitative evidence of the spread of emergent outcomes
• Evidence of the achievement of predetermined outcomes (if these have been articulated)
• Evidence of the ‘average’ experience of participants (or of subgroups of participants) as well as exceptional outcomes
• Information on the views of non-participants and other ‘victims’ of the program
• Improved knowledge with regard to the logic of the program intervention
• Evidence of whether desired outcomes have been achieved, in what situations and why.

Advice for using MSC

MSC is not a quick method. It takes time and an appropriate project infrastructure to generate understanding and value clarification (identifying what people think is important). The full MSC process involves analysis of stories and sharing with both contributors and stakeholders, which requires a programme with several structures in it (for example, local, regional and national project structures) and it needs to be repeated through several cycles.

If you imagine a normal distribution of outcomes for individuals then the stories often come from the extremity of positive change. It can be useful to explicitly add a process to generate and collect stories from the extremity of little or negative change.

Challenges and potential pitfalls

It can be challenging to get engagement of the different groups involved in the process and to maintain their interest. Don't have too many cycles of review. Frequent reporting cycles may lead to a focus on the shorter-term significant changes that can be identified once known cases of longer-term significant change are exhausted