Interventions create distinctive/characteristic patterns of effects. Scriven describes the modus operandi as a set of footprints:
‘Each cause has a set of footprints, a short one if it’s a proximate cause, a long one if it’s a remote cause, but in general the modus operandi is a sequence of intermediate or concurrent events or a set of conditions, or a chain of events, that has to be present when the cause is effective.’
An evaluator can draw on the previous experience of participants and stakeholders to determine what constellation or pattern of effects is typical for an initiative.
The evaluator can build a picture of predicted effects and supplement this picture through research. The combined picture will help the evaluator look for typical patterns of effects and contrasting patterns. The consistency of the ‘trace’ with the predicted pattern helps prove the program theory. Any differences from the predicted pattern helps disprove the program theory or open a new line of questioning.
Realist synthesis can be added to modus operandi. Realist synthesis asks: What mechanisms are known operate with particular subpopulations?
Participants in a leadership programme in the New Zealand public service developed individual development plans. The evaluator articulated the modus operandi of the programme as:
- Participant plans identified needed experiences
- Candidates sought these out
- Candidates obtained a more senior position to gain that experience
Advice for CHOOSING this option (tips and traps)
- Provide the evaluator with the original program theory - if there was one - or relevant documents to develop a retrospective program theory.
Advice for USING this option (tips and traps)
- Develop a list of possible alternatives, based on evidence and information gathered throughout the evaluation process. Test the program theory and counterfactual with participants.
Liaise with the evaluators to check the program theory is correct and the validity of possible alternative explanations and the counter-factual.
- Outcomes, impacts & causal attribution: The example from Jane Davidson's presentation available at the Auckland Anzea symposium in 2010, gives a good overview of causality and modus operandi.
- Advocacy Impact Evaluation: This case study uses the modus operandi approach to investigate the potential influence of judicial advocacy efforts targeted at the Supreme Court decisions.
- Ascertaining Causality in Theory-Based Evaluation: This chapter explores the relative strengths and weaknesses of program theory as a tool for inferring causality.
Davidson, J. (2010, May). Outcomes, impacts & causal attribution. Presented at Anzea regional symposium, Auckland, New Zealand. Retrieved from http://realevaluation.com/pres/Causation-anzea2010.pdf