Consider the implications of the resources available and specific constraints
Identify the resources that can be used for the evaluation, and any particular constraints for them.
The following potential resources could be used for the evaluation:
- Funding to engage external individuals or organizations to design and/or conduct the evaluation or review the design and the final report
- Staff time to either conduct the evaluation or to manage an external contractor
- Time and goodwill of other stakeholders who will be involved in the evaluation – such as partner organizations, community members.
- Existing data
Identify any particular constraints for the evaluation such as:
- Short time before findings are needed to inform decisions
- Poor reputation of evaluation due to previous experiences
- Difficulties in engaging particular groups or in working collaboratively
- Missing baseline data
- Difficulties in observing or getting data about implementation or results – for example, when it is being implemented in remote locations, or in fragile, conflict-affected areas.
- Disagreement about what success looks like – for example:
- Disagreement about the overall goals – for example, is an early childhood program primarily about improving workforce participation of parents or about early learning of children?
- Disagreement about the criteria that should be used – for example, is good research technically very accurate or produced in time to inform an important decision? Is the goal to improve the average health and wellbeing in a community or to ensure everyone is above the minimum requirement?
- Diagreement about the standards that should be used - for example, is a 10% increase in published research a good result?
Do an estimate of the costs to collect and analyse the data, as well as the project management and reporting time needed. If available resources are not adequate for the design, adjust the design and/or resources.
When reducing costs it is essential to consider the implications and how to manage these risks. Some possible options for reducing costs are shown below, along with some possible implications and ideas for managing the risks.
- Reduce the number of Key Evaluation Questions
- Possible implications: Evaluation might no longer meet the needs of the primary intended users
- How to manage these risks: Carefully prioritise the KEQs. Review whether the evaluation is still worth doing
- Reduce sample sizes
- Possible implications: Reduced accuracy of estimates
- How to manage these risks: Check these will still be sufficiently credible and useful through data rehearsal using interval estimates
- Make more use of existing data
- Possible implications: Might mean that insufficiently accurate or relevant data are used. The cost savings might be minimal if they are not readily accessible.
- How to manage these risks: This is only appropriate when the relevance, quality and accessibility of the existing data is adequate – need to check this is the case before committing to use them
- Embed data collection in program implementation
- Possible implications: Might lead to a reduction in data quality
- How to manage these risks: Ensure staff are trained and motivated to collect data properly and have sufficient time and equipment to do so
- Use fewer waves of data collection, including possibly retrospectively created baselines
- Possible implications: Will increase the risk of inaccurate data
- How to manage these risks: Check that retrospective baselines will be sufficiently accurate and that less frequent information on progress will be sufficient to inform decisions
'Consider the implications of the resources available and specific constraints' is referenced in:
- Manager's guide to evaluation :