Making rigorous causal claims in a real-life context: Has research contributed to sustainable forest management?

This article presents an example of a rigorous non-counterfactual causal analysis that describes how different evidence and methods were used together for causal inference without a control group or comparison group.  It includes discussion of whether some contributions of the project could be qualified as necessary for the outcomes and explores potential generalisability of the findings.  The examples brings together contribution analysis, process tracing and realist evaluation, using techniques such as identifying and ruling out alternative explanations and looking for "signature" processes that indicate what has produced outcomes.

This resource and the following information was contributed by Patricia Rogers.

Authors and their affiliation

Thomas Delahais and Jacques Toulemonde.

Year of publication

2017

Type of resource

  • Discussion paper​

Key features

This provides a detailed example of using rigorous non-counterfactual causal analysis – that is, doing an impact evaluation without a control group or a comparison group.  This example explains how a number of methods were used together; a theory-of-change (program theory); contribution analysis; process tracing; modus operandi (or signature)

Who is this resource useful for?

  • Advocates for evaluation
  • Commissioners and managers of evaluation
  • Evaluators
  • Those involved in evaluation capacity strengthening

How have you used or intend on using this resource?

I haven’t yet used the resource  but plan to add it to my set of examples to use in evaluation capacity strengthening.  It would be ideal to develop some teaching support material around this case.  I think people learn a lot from engaging with real data from real cases.

Why would you recommend it to other people?

While there is increasing recognition, at least in some organisations, that it is not always possible to create a credible counterfactual (an estimate of what would have happened in the absence of a program – usually by creating a control group or a comparison group), and there is information about alternatives, there are few good examples available for people to both learn from and use to demonstrate their feasibility. 

This example could be used in evaluation training and advocates for a broader range of designs for impact evaluation as an example of a rigorous impact evaluation without a counterfactual, by evaluators providing more details about how to operationalise non-counterfactual options and approaches; and by commissioners and managers of evaluation as an example by which to hold accountable consultants who promise to produce a rigorous impact evaluation without a counterfactual.

Source

Delahais, Thomas, and Jacques Toulemonde. "Making rigorous causal claims in a real-life context: Has research contributed to sustainable forest management?." Evaluation 23, no. 4 (2017): 370-388.

0
No votes yet
Rate this Resource:
A special thanks to this page's contributors
Resource Suggested By
Director of BetterEvaluation/ Professor of Public Sector Evaluation, Australia and New Zealand School of Government.
Melbourne.

Comments

thomasdelahais's picture
Thomas Delahais
Rating: 
0

Thanks Patricia, we're very happy that you liked this paper. Jacques Toulemonde used this case once, at the European Evaluation Society Event in Dublin in 2014 (the evaluation had just ended then), and I used last year for a workshop on contribution analysis in Québec, organised by the Société Québécoise d'évaluation de programme. The thing is, forest management is not the easiest topic :) (but I guess we don't use CA for simple topics), so we used it alongside another case, which is an evaluation of evaluation for the French government (we have a paper coming on this one). I think the participants had a better view at the end of when and how they could use Contribution analysis in practice!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.