Contribution Analysis is an approach for assessing causal questions and inferring causality in real-life program evaluations. It offers a step-by-step approach designed to help managers, researchers, and policymakers arrive at conclusions about the contribution their program has made (or is currently making) to particular outcomes. The essential value of contribution analysis is that it offers an approach designed to reduce uncertainty about the contribution the intervention is making to the observed results through an increased understanding of why the observed results have occurred (or not!) and the roles played by the intervention and other internal and external factors.
Contribution analysis is particularly useful in situations where the programme is not experimental, i.e. not in trial projects but in situations where the programme has been funded on the basis of a relatively clearly articulated theory of change and where there is little or no scope for varying how the program is implemented. Contribution analysis helps to confirm or revise a theory of change; it is not intended to be used to surface or uncover and display a hitherto implicit or inexplicit theory of change. The report from a contribution analysis is not definitive proof, but rather provides evidence and a line of reasoning from which we can draw a plausible conclusion that, within some level of confidence, the program has made an important contribution to the documented results.
Steps
Six steps are taken to produce a credible contribution story:
1: Set out the attribution problem to be addressed
Determine the specific questions being addressed. Not all cause-effect questions are useful to pursue. Contribution analysis is less suitable for traditional causality questions such as: Has the program caused the outcome? To what extent, quantitatively, has the program caused the outcome? These often are not that useful because they treat the program as a black box and don’t get at the fact that there are usually many causes involved. Contribution analysis is more appropriate for contribution questions: Has the Program influenced the observed result? Has the program made an important contribution to the observed result? Why has the result occurred? What role did the intervention play? and for management questions: Is it reasonable to conclude that the program has made a difference? What does the preponderance of evidence say about how well the program is making a difference? What conditions are needed to make this type of program succeed?
2: Develop a theory of change and risks to it
Develop the program logic/results chain describing how the program is supposed to work. Identify as well the main external factors at play that might account for the outcomes observed. Based on the results chain, develop the theory of change upon which the program is based. This theory of change should lead to a plausible association between the activities of the program and the outcomes sought. The theory of change must include the assumptions made in the results chain and the inherent risks as well as external influences such as donor pressure, influences of peers and resourcing levels. Some links in the theory of change will be fairly well understood or accepted. Others will be less well understood, contested or subject to significant influence other than from the program. In this way you acknowledge that attribution is indeed a problem
3: Gather the existing evidence on the theory of change
It is useful to first use existing evidence such as from past related evaluations or research, and from prior monitoring, to test the theory of change. It sets out the intended results (outputs, intermediate and end outcomes). What evidence (information from performance measures and evaluations) is currently available about the occurrence of these various results? The links in the theory of change also need to be assessed. What evidence currently exists on the assumptions and risks behind these links? Which are strong (good evidence available, strong logic, or wide acceptance) and which are weak (little evidence available, weak logic, or little agreement among stakeholders)? What evidence exists about the identified other influencing factors and the contribution they may be making?
4: Assemble and assess the contribution story, or performance story, and challenges to it
With this information, you will be able to assemble your contribution story that expresses why it is reasonable to assume that the actions of the program have contributed (in some fashion, which you may want to try and characterize) to the observed outcomes. Now you have to assess it. How credible is the story? Do reasonable people agree with the story? Does the pattern of results observed validate the results chain? Where are the main weaknesses in the story? There always will be weaknesses. Weaknesses in the story point to where additional data or information is needed.
5: Seek out additional evidence
Having identified where the contribution story is less credible, additional evidence is now gathered to augment the evidence in terms of what results have occurred, how reasonable the key assumptions are, and what has been the role of external influences and other contributing factors. Augmenting evidence can include the collection of additional, new data such as from surveys, field visits, administrative data, focus groups, national statistical data, etc. as well as the synthesis of evidence from other research and evaluations.
6: Revise and, where the additional evidence permits, strengthen the contribution story
With the new evidence, you should be able to build a more substantive and so more credible story, one that a reasonable person will be more likely to agree with. It will probably not be foolproof, but the additional evidence will have made it stronger and more plausible.
Using a generative perspective on causality to infer that a program made an important contribution to an expected result that has been observed, contribution analysis argues that a reasonable contribution causal claim can be made if:
- There is a reasoned theory of change for the intervention: the key assumptions behind why the intervention is expected to work make sense, are plausible, may be supported by evidence and/or existing research, and are agreed upon by at least some of the key players.
- The activities of the intervention were implemented as set out in the theory of change.
- The theory of change—or key elements thereof— is supported by and confirmed by evidence on observed results and underlying assumptions—the chain of expected results occurred. The theory of change has not been disproved.
- Other influencing factors have been assessed and either shown not to have made a significant contribution or their relative role in contributing to the desired result has been recognized.
Issues
Some issues might arise when taking this approach with regards to:
- Reducing uncertainty about the contribution the intervention is making to the observed results.
- Inferring causality in real-life program evaluations.
- Confirming or revising a programme’s theory of change – including its logic model.
Risks and assumptions are labelled as [O] over which the intervention has no or very little influence, or [I], where the intervention can (should) have an influence, direct or indirect, or [C] where the intervention should be able to directly control.
Resources
Guides
- Addressing attribution through contribution analysis: using performance measures sensibly. This article from the Canadian Journal of Program Evaluation outlines the key steps involved with using contribution analysis as a tool to discover the contribution a program has made to particular outcomes.
- Contribution Analysis: An approach to exploring cause and effect: This brief from the Institutional Learning and Change Initiative(ILAC) explores contribution analysis and how it can be used to provide credible assessments of cause and effect.
- Mayne, J. (2011). Addressing Cause and Effect in Simple and Complex Settings through Contribution Analysis. In Evaluating the Complex, R. Schwartz, K. Forss, and M. Marra (Eds.), Transaction Publishers.
- Mayne, J. (2011) Contribution Analysis: Addressing Cause and Effect in Evaluating the Complex, K. Forss, M. Marra and R. Schwartz (Eds.), Transaction Publishers; Piscataway, New Jersey.
Source
- Mayne, J. (2008) Contribution Analysis: An approach to exploring cause and effect, ILAC methodological brief, available at https://web.archive.org/web/20150226022328/http://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf (archived link)
Comments
The journal Evaluation had a Special Issue in 2012 on Contribution Analysis, Vol 18, #3, July
I'd appreciate hearing about applications of CA john.mayne@rogers.com
See a practical example here, used in a trade union context. We have used same Excel based combination of explanation finder and evidence databases for several other situations. Mail me to get an example of such Excel tool.
https://www.researchgate.net/publication/316102155_Contribution_analysis_in_a_trade_union_context
An updated link to the ILAC brief: http://hdl.handle.net/10568/70124
Hi Shelby,
Thanks very much for sharing the link -
Best,
Alice
BetterEvaluation Website Coordinator
Hi Emmanouil, you can use CA for anything as long as there is a clear casual relation to start with. E.g. effect of bureaucratic administration on change X or y. Regards, Wouter
Wouter is right. You just need to build a decent theory of change model of the causal relationship, which of course may be a challenge!
Permit me some scepticism... How is the above 6 step process any different from what might be called a generic approach to theory testing? What are the unique features of CA?
I am tempted to answer my own question: the very fuzzy description of the sort of claims that would be an acceptable result of such analysis.
But feel free to point out the error of my ways :-)
Is searching for alternative theories enough?
If there is only one known or expected instance of x causing y, then searching and testing alternative theories for tat instance makes sense. But if more than one instance is possible then I suggest other steps are needed beforehand i..e searching for these kinds of instances first:
If there is more than one instance in each above categories then the next step that is needed is careful and justified instance selection.
Among the TPs, instances should be found which are most dissimilar. If the theory that works for both then this is strong support for the theory (if it is the favored theory, or other)
Among the FPs and FNs, the selected instance should be as similar as possible to the TP instance in respect to all other attributes other that those present in the theory conditions (FP), and theory outcome (FN) respectively
Only after careful selection of instances to be compared should effort then go into identifying and testing alternative theories
Hi everyone. I am looking for excel spreadsheet on Byesian updating for contribution tracing. If anyone has it, please share it with me. I think you very much in advance.
Best regards
Naeem
Hi Naeem
Have a look at this blog posting of mine on the subject. Let me know if there are any bits that don't make sense
http://mandenews.blogspot.com/2019/11/using-bayesian-probability-measures-to.html
The middle section on process tracing application will be the most useful but do read the preceding section
regards, rick
Add new comment
Login Login and comment as BetterEvaluation member or simply fill out the fields below.