Clearing the fog: New tools for improving the credibility of impact claims

This IIED Briefing Paper shows that the methods of process tracing and Bayesian updating can facilitate a dialogue between theory and evidence that allows for the assessing of the degree of confidence in ‘contribution claims’ in a transparent and replicable way.

The information provided was supplied by Barbara Befani. 

Authors and their affiliation

Barbara Befani (Independent Evaluation Researcher/Consultant), Stefano D’Errico (monitoring, evaluation and learning lead at IIED), Francesca Booker (researcher in the Natural Resources Group at IIED), and Alessandra Giuliani (researcher in the Natural Resources Group at IIED)

Key features

This IIED Briefing paper discusses the broad consensus that the traditional counterfactual analysis leading to the assessment of the net effect of an intervention is incapable of capturing the complexity of factors at play in any particular policy change. The authors suggest instead that evaluations focus on establishing whether a clearly defined process of change has taken place, and improve the validity and credibility of qualitative impact statements.

Using IIED research in Uganda as an example, it shows that the methods of process tracing and Bayesian updating facilitate a dialogue between theory and evidence that allows for the assessing of the degree of confidence in ‘contribution claims’ in a transparent and replicable way.

How have you used or intend on using this resource?

As far as I'm aware of, the resource provides the clearest link I have seen between Process Tracing and the logic of Bayesian Updating, together with a practical understanding of how this logic is used to identify Process Tracing tests as Hoop test, Smoking Gun, etc. which is sometimes difficult to do in practice.

It's also useful because it provides a clear example of how a contribution claim can be formulated as a process and each of its steps linked to specific pieces of evidence. This is not presented only in theory, but as it was applied to a specific evaluation of policy influence.

Why would you recommend it to other people?

The people who have read the paper have said it was enlightening because it shows that the strength of evidence can be "measured" using the notion of "probative value" (used in law, medical diagnosis, crime investigation, etc.) and presents a clear way to do it in an evaluation.