How does the commissioning process hinder the uptake of complexity-appropriate evaluation?

Drawing on interviews with 19 UK evaluation commissioners and contractors, this paper investigates the role of evaluation commissioning in hindering the take-up of complexity-appropriate evaluation methods and explores ways of improving this.

This resource and the following information was contributed by Patricia Rogers.

Authors and their affiliation

Jayne Cox (Brook Lyndhurst) and Pete Barbrook-Johnson (University of Surrey)

 

Year of publication

2020

Type of resource

  • Discussion paper

Key features

This paper is a useful review of relevant literature on procurement of research and evaluation relevant for policy and practice, summaries of issues raised and possible ways of addressing, and conceptual frameworks.

Many evaluations are undertaken through a competitive tendering process (such as a Request for Proposal) to procure an external evaluator or evaluation team.  Over the years there have been many concerns raised about the ways in which procurement processes can constrain the ability of evaluators to produce good quality evaluations.  Ineffective procurement processes can seriously limit the pool of available evaluators (through overly short timelines, onerous application processes, or inappropriate selection criteria or weighting), and especially reduce the ability for an appropriate evaluation design (many evaluations are explicitly or implicitly designed by those commissioning it, not always with the benefit of sufficient expertise).  Whether or not the expected budget should be disclosed, and the consequences of this, have also been the topic of much debate.  These issues are particularly important for evaluations which go beyond simple designs to address complexity.

Who is this resource useful for?

  • Commissioners/managers of evaluation;
  • Evaluators;

How have you used or intend on using this resource?

I intend to use this paper to consider advice to organisations that commission research and evaluation in areas where complexity-appropriate evaluations are relevant.

Why would you recommend it to other people?

It addresses an important issue in an informed, thoughtful and empirical way.

Source

Cox, J., & Barbrook-Johnson, P. (2020). 'How does the commissioning process hinder the uptake of complexity-appropriate evaluation?' in Evaluation. Retrieved from https://doi.org/10.1177/1356389020976157

0
No votes yet
Rate this Resource:
A special thanks to this page's contributors
Resource Suggested By
CEO, BetterEvaluation.
Melbourne.

Comments

Anonymous's picture
David Brous
Rating: 
0

This is an important article and should be compulsory reading for commissioners of evaluations and their procurement personnel.

With over 30 years in consulting to the public sector in Australia, I have been constantly frustrated by the poorly conceived, regulation determined and protocol focussed briefs that are issued by government agencies at all levels.  There are exceptions, where agency personnel attempt to forge a dialogue with potential providers of evaluation consulting services. However, the norm is the provision of a generic brief, top heavy on procedure and probity and often unable to clearly articulate the context and the challenges of a prospective evaluation.  

The inability to establish a dialogue with a proposal manager and a need to submit basic questions in writing on matters such as data availability and integrity, preferred approaches, potential audiences and the plethora of other matters affecting the preparation of a coherent brief, forces evaluators who are not paid to prepare a losing proposal, to assess their chances to be heard by a potential client. This is particularly thye case when the response comes from a procurement manager with little understanding of evaluation or the project to which it is to be applied.

Selection is now 'off the paper' with no opportunity to present an evaluator's credentials in a face-to-face environment, leading to the development of a dialogue that maximises the strength of the methodology that ultimately meets the client's objectives.

Replication of this British research in an Australian environment and wide dissemination of the results is required as a matter of urgency.

January 18, 2021

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.