Earlier this year I co-taught a course on Evaluation for Public Sector Managers with the Australia and New Zealand School of Government (ANZSOG), using BetterEvaluation to structure the course and as a key resource. In one of the sessions, we generated with participants useful strategies, processes or tasks for managing an internal or external evaluation. The results provide a useful checklist for anyone commissioning an evaluation in the public sector, but is also useful for those working in other sectors.
Obtaining a mandate to evaluate
- Secure appropriate approval within the organisation to do the evaluation.
- Be explicit about the purpose and scope of the evaluation – including levels of involvement of internal staff, how best to present lessons learned and considering need for knowledge translation.
Establishing governance and/or technical advice functions
- Establish an appropriate governance group or reference group. Make sure to identify any conflicts of interest of potential members and determine the appropriate response in case such issues arise.
- Be clear on its function – Advice on technical aspects of the evaluation? Assess progress on evaluation development and implementation?
- Be clear on its decision-making role – Recommend or approve? Veto power on nothing, some things, everything?
- Think about the skills and knowledge required as well as organisational role, and check availability.
- Are members chosen as an individual or as a representative of a group or organisation? If the latter, determine and document expectations and processes for communicating with the larger group.
- Identify appropriate communication processes between the group and the evaluator/evaluation team. Can members contact the consultant(s) directly or only through the Chair of the group?
Preparing a tender or quote
- Establish a timeline that is realistic in terms of likely availability of consultants and existence of primary data sources. Include adequate time for report writing and review (if a formal report is required).
- Identify milestones to assess progress and specify deliverables (e.g., process and content, reports, datasets for in-depth review or later re-analysis)
- Factor in time to manage human research ethics approvals and an iterative process to secure final approval (if this is a requirement of either of the commissioning agency or the evaluation provider).
- Identify source documents and data to refine the evaluation scope and secure access to the consultants – include information in the tender about what is/ is not available.
- Identify budget availability, including contingency and procurement rules. Remember to including IP requirements
- using a ‘Reverse Brief’– the consultants respond to the evaluation brief, pitch proposals and options, reflect what they are hearing, demonstrate their understanding, and identify alternatives including challenging assumptions made. You can consider using a staged process starting with a call for Expressions of Interest -> shortlist against the criteria deemed important -> invite shortlisted applicants to reverse briefing (and possibly provide some funding for this); or,
- Engage evaluation experts to help develop or refine the scope of the evaluation of the evaluation design through a variable contract or separate contract.
- Whatever approach you take, it is advisable to include the budget range or budget limit. You can invite tenderers to provide scalable options.
Preparing the contract
- Append a standard contract to the formal request for tender (RFT) and ask the consultants to identify any areas where negotiation is required. Doing this upfront, will save a lot of time and avoid confusion (or worse) later.
- Analyse risks and develop a risk management strategy.
- Incorporate a process to deal with any changes to requirements.
- Include a fixed fee for deliverables and negotiate the schedule of payments.
- Include any arrangements for sub-contractors including necessary approvals and compliance with contract conditions (e.g., confidentiality).
Other important strategies to consider
- Identify the communications strategy (what, when and how) for the wider organisation including senior management and other staff.
- Ensure contract requirement such as confidentiality agreements, insurance certificates, sub-contractor CVs and approvals are actually monitored and audited; and
- Identify, in advance, any potential ethical issues including processes to follow in particular situations.
So far, we focused on using external evaluators. However, many of the strategies above also apply to internal evaluations albeit less formalised. You may specifically want to consider:
- Internal evaluations are likely to need a need a champion at a senior level. This helps ameliorate the greater risk of the evaluation being shelved/downplayed because the processes are not as formalised as with external evaluators. This is also important in obtaining necessary resources as these are often not pre-identified when the evaluation is conducted internally only.
- It is useful to conduct a skills audit. This can identify any gaps and provide an opportunity to develop strategies to address them. This could involve outsourcing a specific component of the evaluation (where internal skills are lacking) or arranging tailored capacity development (such as through training or mentoring).
- Although a formal contract is not required, it is sensible to formalise internal arrangements in an agreement that covers resources and processes to ensure there are no surprises. Remember that there is an opportunity cost in using internal staff – it isn’t ‘free’.
- Also think through resource implications beyond the evaluation team, such as asking other staff to review and comment on draft documents.
- Ensure there is a contingency plan in case staff with a key role in the evaluation leave the organisation.
- Make sure to document all processes and catalogue all data.