How do we use advisory groups effectively in evaluation?

marlene laeubli loud's picture 17th April 2018 by marlene laeubli loud

Marlène Läubli Loud is an evaluation consultant whose co-edited book (with John Mayne) Enhancing Evaluation Use: Insights from Internal Evaluation Units includes discussion of evaluation advisory groups. Marlène has also  contributed to BetterEvaluation by revising the option page on Advisory Groups.

In my (long) experience in evaluation, I’ve been involved in a number of situations where advisory groups have worked very well, and a number where they haven’t. While these days engaging an advisory group is becoming usual practice, there’s still work to be done to find ways to get the most out of advisory group arrangements.

This blog aims to start a discussion about what advisory group practices work well in what situations. I look back on my experiences and outline some of the conditions that I believe have contributed to securing the “best value” from advisory groups. I’m hoping you will join in and share your own thoughts, experiences and opinions so that we can share ways of engaging and utilising advisory groups to their full advantage. 

What is an advisory group for evaluation?

An advisory group provides advice to those who are making decisions about an evaluation, an evaluation system or the evaluation function - which might be an evaluation steering committee, or an identified manager or management group.  (for more detail, please see the updated page on Advisory Groups which I have contributed revisions to).

Advisory groups are set up to engage the multiple perspectives, values and subject-specific knowledge of key stakeholders and primary intended users. This expert advice can include technical “know how” or contextual interpretation, perspectives, and thoughts.  It can be drawn on throughout the life of the evaluation, from design, through data collection and analysis, to reporting and use.

This can develop a sense of ownership of an evaluation amongst key stakeholders, and also give the evaluation team access to a valuable source of knowledge and perspectives that they wouldn’t otherwise have had.

What constitutes good practice from my perspective

  • Include representation of the key, external stakeholders.
  • Allow the advisory group to play an equal role with commissioners in advising on the evaluation design and process as well as the final results.
  • Ensure that the advisory group’s purpose, organisation and tasks are transparent and acceptable to all members from the onset (with perhaps reminders throughout to be sure!)
  • Establish a collaborative working atmosphere.

For instance, wherever advisory groups have been set up by commissioners who are not fully convinced of working in partnership with the stakeholder, the purpose, modus operandi and tasks of the advisory group are likely to be unclear. This can lead to misunderstandings, mistrust, tensions and unresolvable conflicts. At worst, the advisory group is just a means of ticking the box rather than being used as a valuable resource!

  • Acknowledge the extra workload and time input involved

An advisory group inevitably means an additional workload for the evaluators and those managing the evaluation. So it is vital to check that the needed support is secured both by the relevant commissioner and advisory group organisations (for the evaluation manager) and within the evaluation budget (for the evaluator).

  • Present and respect stakeholders’ multiple values perspectives equally – make sure no one or two stakeholders are allowed to sabotage the event.
  • Ensure all advisory group members work together in agreeing on a communication strategy for dealing with, and acting on, the evaluation results.
  • Develop an agreed process for resolving conflicting views and recommendations.

For example, unresolved disagreements can be either written up as a preface to the final evaluation report or into the Executive Response.

Under such conditions, with the inclusion of stakeholders, the evaluation enjoys invaluable expertise and input. The validity/credibility of the evaluation and its use is better assured and - equally important - the status of evaluation is likely enhanced.  One more thing! Recognised or not, with real engagement and commitment, the whole process is in itself a learning process too! For example, determining the theory of change together is a real learning event.

Particular challenges

Should intended beneficiaries be included in the advisory group?

Beneficiary representation would be a plus where this is feasible and would make sense, but this can be difficult to do in practice. In my experience, I have only seen this happen on two occasions.  Who you include in terms of beneficiary representation, and how you do this, can be a challenge. For example, to include the public, consumer associations could be used as representatives, but there are often more than one.

Moreover, for a country like Switzerland, for instance, there are four official languages, each with its own consumer association. One alone would not be representative. Issues like these need to be carefully thought through before putting together the advisory group.

Should external evaluation experts be included in an advisory group or a separate technical advisory group?

 I have found that mixing evaluation methodology expertise together with stakeholders in an advisory group doesn’t work! Discussing the “nitty gritty” of evaluation theory and practice is best kept separate. But, of the few cases I know of where a separate methodology advisory group was set up to provide peer support/advice to the evaluator(s), only in one case would I say it was a success. Understandably, once an evaluator is hired, mutual trust needs to be secured between the evaluator and the commissioner and its internal evaluation manager. Additional methodological “outsiders” are often not appreciated!

However, in the one exception that I mentioned, each of the key stakeholder organisations assigned a manager from their own internal evaluation unit to meet together with the evaluation team. From the onset, there was a collaborative spirit established, with everyone pulling together to find solutions to the somewhat difficult challenges of dealing with an intervention with widespread, global coverage. And it worked! (Probably the legitimacy of the group helped considerably).

Your experiences?

So how much of the above rings true for you? What are your ideas, thoughts, experiences on these or other issues relative to the use of Advisory Groups? What's worked well for you in the past? Do you have comments on the good practices I have suggested – or answers to the particular challenges? Let us know in the comments below.

Further reading

Option page

Advisory Group

An Advisory Group can be established to provide advice on an individual evaluation, a series of evaluations, or the evaluation function within an organization.

Resource

Enhancing Evaluation Use: Insights from Internal Evaluation Units

This book, co-edited by Marlène Läubli Loud and John Mayne, offers invaluable insights from real evaluators who share strategies they have adopted through their own experiences in evaluation. It discusses a number of challenges, solutions, and lessons drawn from the experience of evaluators working in a wide range of organizations. Contributors discuss factors that help or undermine attempts to foster an evaluative thinking and learning culture within an organization.

 

A special thanks to this page's contributors
Author
Dr DPhil, LAUCO, Evaluation & Training.
Rossens, Fribourg, Switzerland.

Comments

Anonymous's picture
Bassirou Diagne

Bonjour Dr DPhil, LAUCO,

Je trouve intéressant votre édition et très riche vos expériences dans ce domaine. donc en pratique, tous utilisent directement un groupe consultatif ou indirect avec la restitution finale des résultats d'évaluation. Ainsi, vous pouvez inviter les bénéficiaires à participer à ce groupe de sorte que l'erreur est leur participation dans les commentaires des résultats. de vos expériences comment les bénéficiaires invités à participer .......

marlene laeubli loud's picture
Marlène Läubli Loud

Bonjour !

Merci de vos commentaires et de votre question. En effet, c'est un grand défi de soliciter la participation des bénéficiares; difficile à identifier une représentation juste / adéquate, problème qu'ils soient trop timides face à face aux autres protagonistes dominantes - chacun avec leur propres intérêts etc. Ainsi, cela dépend beaucoup du contexte, ce que vous cherchez etc. Comme on ne peut pas soliciter la participation de tous les bénéficiares, on doit trouver des représentants dont le défi est d'en faire le bon choix ! Je cite un exemple seulement; pour un programme ayant à faire avec des mesures pour réduire  des problèmes liés à la drogue, nous avons invité une association de parents des toxicomanes emprisonés à prendre part dans la discussion avant-programme et pour la resitution des résultats de l'évaluation du programme. Cela nous a beaucoup aidé à prendre en compte d'autres perspectives et avis.

Mais ce n'est pas évident ! Merci de votre question et bonne continuation !

Marlène

Marlène

Anonymous's picture
j cooper

Great website and totally agree with this article.
For me as head of research and evaluation in a public sector org, the best thing about advisory groups (we usually use the term steering group) is getting that ownership from key stakeholders which is vital for evaluation having an impact. For me the main risk of the group is that the key decision makers do not join the group but instead delegate it to their team who can not always speak for the strategic decision makers. So I think we need to sell the value of evaluation and being in a steering group as vest we can to the key decision makers.
Thanks
Jon

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.