Making rigorous causal claims in a real-life context: Has research contributed to sustainable forest management?

This article presents an example of a rigorous non-counterfactual causal analysis that describes how different evidence and methods were used together for causal inference without a control group or comparison group. 

It includes discussion of whether some contributions of the project could be qualified as necessary for the outcomes and explores potential generalisability of the findings.  The examples brings together contribution analysis, process tracing and realist evaluation, using techniques such as identifying and ruling out alternative explanations and looking for "signature" processes that indicate what has produced outcomes.

This resource and the following information was contributed by Patricia Rogers.

Authors and their affiliation

Thomas Delahais and Jacques Toulemonde.

Key features

This provides a detailed example of using rigorous non-counterfactual causal analysis – that is, doing an impact evaluation without a control group or a comparison group.  This example explains how a number of methods were used together; a theory-of-change (program theory); contribution analysis; process tracing; modus operandi (or signature)

How have you used or intend on using this resource?

I haven’t yet used the resource  but plan to add it to my set of examples to use in evaluation capacity strengthening.  It would be ideal to develop some teaching support material around this case.  I think people learn a lot from engaging with real data from real cases.

Why would you recommend it to other people?

While there is increasing recognition, at least in some organisations, that it is not always possible to create a credible counterfactual (an estimate of what would have happened in the absence of a program – usually by creating a control group or a comparison group), and there is information about alternatives, there are few good examples available for people to both learn from and use to demonstrate their feasibility. 

This example could be used in evaluation training and advocates for a broader range of designs for impact evaluation as an example of a rigorous impact evaluation without a counterfactual, by evaluators providing more details about how to operationalise non-counterfactual options and approaches; and by commissioners and managers of evaluation as an example by which to hold accountable consultants who promise to produce a rigorous impact evaluation without a counterfactual.