Why do we need more real-time evaluation?

We’re currently going through a global period of rapid change and adaption, due in large part to the effects of the COVID-19 pandemic on our lives and work.

As our world is changing, many individuals and organisations are finding that traditional evaluation methods are not meeting the needs of rapidly changing implementation or brand new interventions where interventions are being designed as they are implemented. It’s in this context that it’s become important to use evidence and support learning in real-time.

Real-time evaluation (RTE) has been practised and documented over the past 20 years, initially used primarily in humanitarian projects. There is now increasing interest in learning from this experience to inform evaluations in other areas, especially in development. This is why we have written a working paper on what can be learned from previously evaluation practice to inform current practice.

The term ‘real-time evaluation’ has been used in different ways, and it can be useful to think of these as degrees of implementation, that can include one or more of the following dimensions:

1. real-time (or more current) data collection;

2. real-time (or rapid) reporting;

3. multiple timings of evaluative activity;

4. support for different types of learning (e.g. single-loop, double-loop, and triple-loop); and

5. engaging different users together in dialogue for sense-making and action planning.

While a real-time evaluation can include all of these elements, it could also include only collection of real-time data (that might be analysed and reported later), or the collection of real-time data along with real-time reporting (such as in a dashboard).

Below, we explore in more detail how a real-time evaluation with one or more of the following dimensions can be used to fill the gaps that existing M&E systems may not be able to meet.

1. Real-time (or more current) data collection

When evaluation is not planned and begun from the start, there is often poor documentation of implementation especially in rapidly changing interventions, making it difficult to inform decisions or document lessons for future interventions.

In RTE, data collection is undertaken during implementation and has often used field visits and interviews, or remote methods in more recent times, to document implementation more comprehensively and accurately than would otherwise be the case. This data can be used immediately and in the future for other interventions.

2. Real-time (or rapid) reporting of evaluation data

There’s often considerable delay between data collection and evaluation reporting in non-real-time evaluations. Sometimes there is a debrief before leaving a field visit, but these tend to focus on checking the accuracy of data collection. This can cause problems when existing monitoring systems are not providing timely or relevant data needed to inform actions and decisions.

In RTE, rapid feedback of findings is provided as part of a field visit or engagement, which is useful when there is scope to make changes in response to real-time data.

3. Multiple timings of evaluative activity

Non-real-time evaluation often consists of episodic activity – such as a mid-term and end of project evaluation without systematic use of evaluative evidence in between. This limits the potential for use in improving an existing intervention, especially when currently planned evaluation events or active engagement with monitoring data will be insufficiently frequent to support decisions.

RTE is done at a number of points throughout implementation. The timing can be regular – for example, monthly or quarterly– or linked to particular decisions or activities, such as planning events.  

4. Support for different types of learning – single-loop, double-loop, and triple-loop

Non-RTEs often focus primarily on single-loop learning – identifying discrepancies between planned and actual activities and results and suggesting ways to improve compliance. This is important but not sufficient, especially when interventions are not well understood or need to be adapted to address changing circumstances. Double-loop learning (reviewing and revising assumptions and conceptual models such as theories of change) can sometimes be seen in end of project evaluations, where it is intended to inform scaling up and replication. It is uncommon to pay explicit attention to supporting triple-loop learning (reviewing how learning occurs and should occur in terms of processes and evidence used).

Support for double-loop learning throughout implementation is especially important in cases where adaptive management is needed in the face of ongoing uncertainty, and assumptions need to be revised. Similarly, when there is a clear theory of change underpinning an initiative it can be important to review and revise this throughout implementation.

RTE explicitly addresses all three types of learning – single loop (identifying discrepancies), double-loop (supporting revisiting of assumptions and the implications for making changes to the theory of change and implementation activities), and triple-loop (reviewing what evidence is being used and how to support decision making).

5. Engaging different users together in dialogue for sensemaking and action planning

Sometimes, evaluations are focused primarily on producing an independent judgment and conclusions. Implementers are seen mostly as data sources, and potentially evaluation users, with the external evaluator drawing conclusions and making recommendations that are reported back for uptake.

RTE explicitly involves bringing a range of stakeholders together to make sense of the data and jointly develop recommendations for action, bringing greater expertise to bear, and also developing greater ownership of and commitment to the findings and recommendations.

When is a real-time evaluation not suitable?

While there are real benefits to using an RTE over a conventional evaluation in certain situations, it’s important to be aware that RTE is not always appropriate. There can be potential risks in RTE when chosen in unsuitable contexts or not implemented well – especially in terms of providing inadequate information to inform decision-making.

In particular, the rapid data collection process, and rapid sensemaking and reporting, can increase the risk of drawing invalid conclusions, through inadequate data collection and analysis or through inadequate management of cognitive biases such as confirmation bias, where people tend to notice and remember information that confirms their existing understanding.  We recommend the brief, ‘How shortcuts cut us short: Cognitive traps in philanthropic decision-making’,  by Tanya Beer and Julia Coffman for some specific strategies to manage cognitive biases.

Final thoughts

The situations in which RTE is appropriate, and what is needed to make it work, depend on the way in which it is being defined, and the intended benefits from doing it. In situations where RTEs are feasible and appropriate, RTEs can offer solutions to many of the problems faced with traditional evaluation.

BetterEvaluation’s latest working paper outlines the different ways in which RTE has been defined and is understood to work. It analyses how RTE is similar to and different from other approaches to supporting evidence-informed action. It discusses when it is appropriate to use RTE and what is needed to make it work well. It also provides links to additional resources, further reading and examples of RTEs.

Read more

Monitoring and evaluation working paper series, #4: Real-time evaluation

Feature image: Photo: Jutta Benzenberg/World Bank (2014) Monitoring and Evaluation System, from the World Bank Photo Collection. The World Bank supports the creation of the practical and effective monitoring and evaluation system critical for the success of Romania’s National Climate Change Strategy and Action Plan.