Becoming aware of contradictory demands on evaluation systems

The following article was written Marijn Faling, Assistant Professor Evaluation and Private-sector Development at the International Institute of Social Studies, Erasmus University Rotterdam, the Netherlands.

Contradictory demands in evaluation

Evaluation systems often face seemingly contradictory demands. Different stakeholders have different expectations of evaluation. Implementers may need evaluation systems to conduct monitoring for learning, whereas donors may want to produce evidence on impact for accountability.

But did you know that learning vs accountability is just one among multiple contradictory demands on evaluation systems? Evaluation of complex, multi-actor programmes, set in complex and uncertain environments especially forms a breeding ground for contradictory demands (see the blog post ‘addressing complexity’ for a discussion on complicated vs complex interventions ). Most notably, M&E systems are required to anticipate unknown outcomes and untangle multiple related impact pathways, while remaining actionable, practicable, and cost-effective. For well-functioning M&E systems, we need to be aware of these competing demands.

Overlooking competing demands might jeopardize M&E systems

Let’s take a closer look at why awareness of competing demands is important. First, M&E systems need to meet various requirements, to preserve and safeguard their legitimacy in the eyes of various stakeholders. Second, if overlooked, competing demands may result in tensions for evaluators and stakeholders. This obviously complicates the daily work of evaluation professionals. But more importantly, a lack of clarity and understanding regarding the purpose and design of M&E systems often results in failing M&E systems. Think of a mismatch between program characteristics and evaluation design, or methods that are unable to answer evaluation questions. So, acknowledging and addressing contradictory demands is essential. This informs recognizing, acknowledging, and making choices regarding various design options in M&E, to facilitate the creation of well-functioning evaluation systems.

There are different ways to approach these competing demands, with differing implications. When approached as trade-offs, one demand is preferred over the other. M&E systems may consequently be designed solely for learning, and not accountability. Although this may seem a simple and straightforward solution to avoid complications, it hampers legitimacy as well as functions and uses of M&E systems. Therefore, we advocate to approach contradictory demands as paradoxes. This means recognising competing demands and accommodating them together in a single M&E system.

Five paradoxes in M&E

Based on our experiences with designing, implementing, and managing the M&E system for 2SCALE, a complex multi-stakeholder programme to advance food security through inclusive agribusiness in Sub-Sahara Africa, we identify five paradoxes. By describing the paradoxes, we hope to provide a language to identify, discuss and act upon competing demands. We give insights from this case as illustration of how we have tried to accommodate all into one single system. As such, we do not claim this is the preferred approach. Why not? First, we encountered that any attempt to embrace paradoxes is a temporal solution that leads to new paradoxes and challenges. There is no ultimate solution to competing demands and they therefore require continued attention. Second, although paradoxes may be universal, we expect ways of addressing them to be highly context-specific and dependent on the program as well as the M&E system.

The paradoxes, competing demands, their underlying logic, and our way of dealing with them are included in the table below.

So what?

To summarize, M&E is a complicated endeavor and evaluators often face competing demands. These range from simultaneous expectations about involvement as well as autonomy from the program evaluated, to rigorous and reliable M&E systems that are simultaneously sufficiently flexible to adapt to changing intervention logics. We argue that identifying and accommodating these competing demands helps to ensure legitimacy and success of M&E systems. We don’t think we have identified all paradoxes in evaluation, so let us know if you encounter a paradox in your work!