Week 49: The 1st international conference on realist approaches to evaluation: my ‘realist’ take-aways

Tiina Pasanen's picture 4th December 2014 by Tiina Pasanen

Tiina Pasanen is a Research Officer for the Research and Policy in Development (RAPID) Programme at the Overseas Development Programme (ODI). In this blog, Tiina shares her top three realist ‘take-aways’ from the 1st International Conference on Realist Approaches to Evaluation and reflects on when or how realist evaluation may be most useful.

Raise your hand if you are a) a positivist b) a realist c) a constructivist, or d) somewhere between these categories. This was what we were asked to do during one of the keynote sessions at the 1st International Conference on Realist Approaches to Evaluation and Synthesis: Successes, Challenges, and the Road Ahead - an epistemological first for me at an evaluation conference!

Realist evaluation (RE) is a relatively new term - developed by Pawson and Tilley in 1997. It is a member of a family of theory-driven evaluation approaches which seeks to clarify root causes of programme outcomes by evaluating evidence from qualitative, quantitative and mixed-method research.

Hosted by the University of Liverpool, the conference brought together 150 researchers and evaluation practitioners to learn and share experiences on using RE and synthesis for assessing complex evidence for programmes, interventions or policy implementations in health or social sciences field.

Here are my top three ‘realist’ take-aways, as a beginner to the field:

1. Understand the limits of the ‘what works?’ question

It is important to remember that there are always winners and losers in social interventions. A development intervention (especially a complex one) will never work in the same way for all people and there are limits to using statistics of averages without understanding the diversity within the data. With this in mind, RE’s tagline ‘what works, for whom, in what circumstances and why?’ is clearly a better starting point than just ‘what works?’ – which is frequently seen in ‘traditional’ impact evaluations.

2. Pay attention to context and mechanisms

There are multiple ways to understand mechanisms in practice. To use a common metaphor: gravity is the mechanism which causes a ball to drop to the ground, but gravity will act differently in different contexts (e.g. the ball will drop at a slower speed on the moon than on earth). The mechanism is only observable by the effect (i.e. the ball drops). Mechanisms in social development programmes tend to be cognitive or emotional processes, such as trust. RE looks for context-mechanism-outcome hypotheses (configurations), which is one way to capture the complexity of an intervention. First, you have to understand the contextual factors – such as the environment, public policy, or socio-economic conditions – followed by identification of the mechanisms at play, and how they are triggered by the contextual factors.

3. Appreciate multiple sources of evidence

RE claims to be ‘method-neutral’ i.e. it does not impose the use of particular methods. While I question whether this is ever 100% possible (we evaluation practitioners tend to have preferred methods) I very much like the idea of mixing methods, synthesising evidence from multiple sources and trying to make sense of it. Usually, mixing methods, at least in traditional impact evaluations, aims to complement statistical analysis. But according to the RE approach, we are not seeking one ‘representative’ answer, but rather accepting that different results can reveal something about the diverse patterns of impact.

What was missing from the conference?

The conference facilitated thought provoking and timely debate, but there were a few missing dimensions. I was aware of a constant drive to define RE – and especially to distinguish it from randomised control trials, which are seen to represent the positivist approach, which was highly criticised at the conference. Unfortunately, this can keep the divided paradigm discussions alive (something that has been going on in the international development field way too long), instead of creating new possibilities for collaboration. However, by the end of the fourth day I did start to hear an increasing number of more balanced and collaborative comments emerging.

In my work, I’m surrounded by researchers and evaluators who focus on the production, ‘brokering’ and use of knowledge and evidence for policy. I was therefore surprised by the lack of this focus at the conference overall. Perhaps this is to be expected, given that it is a theory-driven approach and the academic environment incentivises the production of peer-reviewed articles over more policy-focused outputs and dialogue with key stakeholders. Therefore, I was pleased, to attend the final conference session, which focused on working with policy-makers.

Reflections on when or how RE can be most useful:

With my new RE lenses, I began to wonder whether researchers collaborating with policy-makers should start with an analysis of the context and mechanisms of policy-making. This could be used to identify what type of evidence and means of communication are required by policy-makers to make evidence-informed decisions in highly fluctuating and political environments. I leave you with my suggestions of where RE could be particularly suitable and useful:

  • When it is more important to understand the mechanisms (reasons) behind the outcomes than knowing the prevalence of the effect.
  • When an intervention has shown diverse patterns of impact and we want to understand why an intervention seems to work for some and not for others.
  • When we want to apply an intervention to another context but are not sure why it works in the first place. Understanding the context and mechanisms behind the outcomes can strengthen the implementation of the programme to other contexts.

More on information Realist Evaluation:

BetterEvaluation website
The RAMESES (Realist and Meta-narrative Evidence Syntheses: Evolving Standards) project
Realist Impact Evaluation: An Introduction

A special thanks to this page's contributors
Author
Research Fellow, Overseas Development Institute.
United Kingdom.

Comments

Gill Westhorp's picture
Gill Westhorp

Hi Rick and Tiina

The sad truth is, there isn't a single agreed definition of the term 'mechanism' in realist evaluation.  To go back to the roots: the use of the term is drawn from realist philosophy. One of the lead realist philosophers, Roy Bhaskar, talked in terms of three 'strata' of reality - the real, the actual and the empirical. To over-simplify, the empirical can be observed/experienced; the actual exists whether we observe it or not, and includes but is not limited to empirical; and the real includes that which cannot be observed but causes that which can be observed, as well as the actual and the empirical. My favourite shorthand for this is "that which has real effects, is real" - so for example, gender is not material, and is different in different cultures, but is nonetheless real because it has real effects.

This is why realists talk about mechanisms as existing 'at another level of reality' and as being 'not able to be observed'.  So one might summarise the philosophical notion of mechanism as being 'that which causes but which cannot be directly observed'.  (Note that this doesn't mean that mechanisms can't be investigated - they can, but because they can't be observed, they have to be theorised first.)

The phrase 'reasoning and resources' comes from Pawson and Tilley's first book.  They were translating this notion of 'that which causes but cannot be observed' in relation to what programs are trying to do (ie cause a change of some kind.) They noted that programs may provide 'the same' resources or opportunities to their participants, but nonetheless achieve different outcomes.  For a realist, this means (de facto) either that mechanisms have only 'worked' for some, or that different mechanisms have operated for different people.  The different outcomes, they argued, happen because participants 'reason differently' in response to the resources on offer.  The participants' reasoning (which we might note, can't be observed but can be investigated) underpins their decisions, which underpin their actions, which generate outcomes (or not).

However, the moment one starts to think about a real program, the apparent simplicity of the 'reasoning and resources' heuristic reveals its actual complexity. Imagine a teacher and a group of students. The teacher 'teaches' - but what exactly is the resource? For some students: the teacher provides new information which some students learn, with the outcome of 'learned information'. We can call that a mechanism - but note that different students in the same class may learn it to different degrees -some just learn by rote, some to the extent that they can manipulate it, some not at all. Some might learn all of it, some only some of it. This gives rise to a couple of the basic outcome questions in realist evaluation: 'to what extent' and 'in what respects'. For those who don't learn anything at all - that might be because they didn't have necessary prior learning to join the new learning on to, or because they in fact already knew it. Those are two quite different mechanisms, but with the same (non-learning) outcome. 

For other students, the 'resource that matters' might actually be the encouragement the teacher provides (external resource), which taps into a tentative student's courage (intrinsic resource - some might call this a mechanism in its own right) to give something a go, and when they succeed at least in part, that experience of success operates as a feedback loop which encourages further effort, which over time generates enhanced self-efficacy. This too is a mechanism - regardless of whether the student ever mastered the information in the original lesson.  For another group of students, it's actually other students that 'get the message through' to them (whether or not the class was set up as peer education) - for them, the mechanism that achieves the learning outcome is something to do with peer learning - perhaps due to trust, but perhaps due to different explanatory techniques used by the 'student-teachers', or because the 'student-teacher' did something in a different way from the teacher and the 'student-learner' observed it and understood it.  Until you understand what it is that matters about the peers, what they do, their relationships with the 'student-learners' and how the 'student-learners' 'reason' in response to the resource(s) their peers provide, you won't really know how to make it happen again (ie how to set up an effective peer education program).

This example I hope demonstrates the difficulty with naming program activities as 'resources'.  The program activities may include or provide many kinds of resources, with each kind of resource triggering different 'reasoning' for different participants, and the outcomes being dependent on the actual mechanisms that 'fire' (operate, happen, work). 

A couple of quick final notes: 'reasoning' can include anything that happens inside the participant's head: reasoning in the sense of 'logic in use', or changes in affect (emotion), or belief systems.  A program may 'trigger' existing 'reasoning' or introduce new 'reasoning' - intended or unintended. All of these can be mechanisms. None of them will work for all participants or in all places.

Secondly - mechanisms don't just fire at the 'participant' level of a program; they may operate at any stage of a program's implementation. Imagine that the Education Department has introduced a program to change the way teachers' teach, with a view to improving student learning outcomes.  There will be decisions required at central departmental level, at program management level, by principals and other leaders in schools, by teachers... and finally by students, and perhaps by their parents.  Each 'decision-point' hides a 'mechanism point' (ie a point where multiple mechanisms may fire and which may support, enable or undermine the eventual effectiveness of the program).

Thirdly - not all realists agree, but many (myself included) go back to the original notion of mechanisms in realist philosophy to argue that there are different kinds of mechanisms that operate at different levels of systems. There are mechanisms that operate in the material world (gravity causes the ball to fall to the ground); mechanisms within the individual (as described in the above example); mechanisms that operate at the group level (eg those that operate in the relationships within groups - 'group think', mass hysteria); at the institutional level, and at the broad societal level. Different kinds of programs seek to trigger (or disable), or introduce mechanisms at different levels.

Sorry for the long post but I hope it explains why there's not a single sentence to explain 'mechanism' to newcomers...

Rick - by the way - I sometimes use (and suggest when teaching RE) using CSMO - where the S stands for 'strategy' or 'activity' - so that people can see whether there are particular mechanisms that relate to particular program activities.  It works in some cases, but not when multiple activities/strategies are required in order to fire a particular mechanism...

Cheers to all

Gill

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.