Select an evaluator / evaluation team
A transparent selection process uses explicit criteria and involves more than one person (often a panel or committee) to discuss and agree on the selection.
The selection criteria and process as well as who will sign-off should, ideally, be discussed and agreed early on in the evaluation process (see Step 1 for further information).
Shortlisting potential candidates
Applications need to be reviewed and a shortlist of potential candidates determined. Typically, applications are ranked in terms of the degree to which they match the selection criteria. Ranking is facilitated by carefully considering the relative importance of the selection criteria for the work at hand. In the first instance, use the ‘essential’ and ‘desired’ qualities but further distinctions may also be needed (such as specific content expertise or previous work in a particular context or with certain stakeholders) (see list of evaluator qualities).
Final selection of the evaluator(s)
The shortlisted candidates are then further queried through a follow up interview (face-to-face or by phone/internet) or other communication means. Decide who will be on the selection panel and what questions will be asked. For example, the following questions may be helpful:
- Knowledge and experience – Does the bidder have the market, industry and/or professional experience to meet your agency’s needs?
- Team composition – Are the team members listed in the proposal the people who will actually do the work? Is the team known to your agency?
- Business values and policies – Does the bidder reflect your agency’s business values and expectations of quality?
- Understanding the job – How well does the bidder understand the size and scope of the job? Are they up to the challenge?
- Innovative and creative thinking – Will the bidder be able to respond to emerging issues throughout the evaluation?
[The above list of questions were adapted from: Choosing the right consultant. In: Policy Makers Toolkit. Steps in Managing and Evaluation Project. Premier & Cabinet, NSW Government, Australia.]
The selection of a team leader is particularly crucial to the success of the evaluation. A good team leader is results-oriented and focused on management objectives. She/he must have demonstrated ability to manage and synthesize the input and participation of the various team members as well as the range of stakeholders involved. Good communication skills –both verbal and written– and good facilitation and negotiation skills are essential.
Determining potential conflict of interest or safeguarding an appropriate level of ‘independence’
Another important issue in the final selection is to determine if the individual or their organization has any potential conflict of interest which may compromise the credibility of the evaluation. Criteria for excluding certain individuals/organizations from undertaking the evaluation could include:
- not having been involved in the design, implementation or oversight of the activities;
- not having received funding from the organization / program in the recent past (such as at least not in the last two years)’
- not having a stake in what the evaluation uncovers.
Conducting reference checks
As the final step before hiring an evaluator, look at their past work, check references, and make sure you feel comfortable that they are credible, competent, and capable of leading a useful evaluation process. Some key issues that can be discussed in the reference check are:
- Timeliness – Did they meet the milestones and deliver the product on time?
- Responsiveness – Were they flexible enough to deal with unexpected challenges or delays? Did they respond to the project / program team in an appropriate manner?
- Relevance – Did they follow the terms of reference?
- Professionalism – What was their work style, communication ability, or degree of cultural sensitivity?
- Evaluator good practice – Did they engage openly with the project / program team? Did they proactively explain their decisions based on good practice in evaluation?
- Evidence-based conclusions – Were the report’s conclusions evidence-based or mainly conjecture?
[The above list of questions was adapted from: Willard A. Managing and implementing an evaluation. Guidelines and Tools for Evaluation Managers. USA: American Red Cross, Catholic Relief Services, USAID, 2008.]
- Checklist for selecting an evaluator (pdf download)
W.K. Kellogg Foundation Evaluation Handbook (page 63) , 2004.
- Choosing the right consultant
Premier & Cabinet, NSW Government, Australia.
' Select an evaluator / evaluation team' is referenced in:
- Manager's guide to evaluation :