How does the commissioning process hinder the uptake of complexity-appropriate evaluation?

Drawing on interviews with 19 UK evaluation commissioners and contractors, this paper investigates the role of evaluation commissioning in hindering the take-up of complexity-appropriate evaluation methods and explores ways of improving this.

This resource and the following information was contributed by Patricia Rogers.

Authors and their affiliation

Jayne Cox (Brook Lyndhurst) and Pete Barbrook-Johnson (University of Surrey)

Key features

This paper is a useful review of relevant literature on procurement of research and evaluation relevant for policy and practice, summaries of issues raised and possible ways of addressing, and conceptual frameworks.

Many evaluations are undertaken through a competitive tendering process (such as a Request for Proposal) to procure an external evaluator or evaluation team.  Over the years there have been many concerns raised about the ways in which procurement processes can constrain the ability of evaluators to produce good quality evaluations.  Ineffective procurement processes can seriously limit the pool of available evaluators (through overly short timelines, onerous application processes, or inappropriate selection criteria or weighting), and especially reduce the ability for an appropriate evaluation design (many evaluations are explicitly or implicitly designed by those commissioning it, not always with the benefit of sufficient expertise).  Whether or not the expected budget should be disclosed, and the consequences of this, have also been the topic of much debate.  These issues are particularly important for evaluations which go beyond simple designs to address complexity.

How have you used or intend on using this resource?

I intend to use this paper to consider advice to organisations that commission research and evaluation in areas where complexity-appropriate evaluations are relevant.

Why would you recommend it to other people?

It addresses an important issue in an informed, thoughtful and empirical way.

Sources

Cox, J., & Barbrook-Johnson, P. (2020). 'How does the commissioning process hinder the uptake of complexity-appropriate evaluation?' in Evaluation. Retrieved from https://journals.sagepub.com/doi/10.1177/1356389020976157