Reducing costs is an option to consider if evaluation costs outweigh the predicted benefits or available resources. There are a number of strategies for reducing costs by modifying the evaluation design. Consider these options carefully because most of them involve threats to the validity of the evaluation findings and recommendations.
Often the available resources are less than required to implement your ideal evaluation design. You may need to revise your evaluation to make the best use of resources. Strategies for reducing costs of evaluation planning, data collection, and analysis include:
- Focus the evaluation to eliminate nonessential data collection
- Reduce the frequency of data collection
- Use available data
- Reduce the sample size
- Use cheaper data collection options
1. Focus the evaluation to eliminate nonessential data collection
Clarify the information needed and the decisions to which the evaluation n findings will contribute. In consultation with relevant stakeholders distinguish between “nice to know” and “need to know” information to eliminate nonessential data collection and analysis. Including only essential data can bring savings in design, data collection, analysis and reporting. This may involve revisting the purpose of the evaluation and the key evaluation questions.
2. Change the research design
For example, “eliminating the collection of data on the project or comparison group before the project begins (pretest) or on the comparison group after the project is implemented (posttest). In the simplest design, when data are collected on only the posttest project group, the data collection budget can be reduced by as much as 80%.” Such designs necessarily involve compromises in validity. RealWorld Evaluation (2012) describes 19 impact evaluation designs and ranks them according to their strengths, and their estimated cost savings (pp. 46-47).
3. Reduce the frequency of data collection
Reducing the number of data collection points in a time series (for example, every 2 months instead of every month) will reduce costs. Make sure you get expert advice to check that the data will still be sufficiently robust to provide valid results and be seen as credible.
4. Use available data
Start with the project’s own monitoring records. What information is currently available? How might this information be mined to address evaluation questions? Consult with staff in the field because sometimes they may be collecting or have access to information that is not requested on reports.
If the evaluation’s information needs cannot be met through the project’s own monitoring records, look outside the project at other potential sources:
- Census or survey data covering the project and comparison communities.
- Public services agencies, such as schools, health centers, agricultural extension centers, etc.
- Newspapers and other mass media
- Information from community organizations
- Dissertations and other academic studies
Assess the information from secondary sources carefully to ensure that it can be used to answer the evaluation questions. Ensure that the data sources match the proposed evaluation sample, and that the data be obtained easily and ethically.
5. Reduce the sample size
Reducing the sample size to work within budget is a matter of balancing information needs with technical considerations. First consult with key stakeholders to understand the decisions to which the evaluation will contribute, and the degree of precision appropriate to those decisions. After coming to an understanding of the degree of precision required, consider the factors affecting the sample size for:
- quantitative options
- qualitative options
- mixed-option designs
6. Use cheaper data collection options
This strategy is similar to eliminating nonessential data collection, but also includes substituting time-consuming or expensive options with more economic options. For example, consider:
- INSTEAD OF surveys and interviews USE focus groups and participatory rural appraisal
- INSTEAD OF surveys USE direct observation
- INSTEAD OF surveys of a larger sample USE interviews or surveys of a few key informants
- INSTEAD OF evaluator-administered questionnaires USE self-administered ones
Advice for CHOOSING this option (tips and traps)
Consider each of the five cost reduction strategies carefully because most of them involve threats to the validity of the evaluation findings and recommendations.
Keep in mind: “With emergent designs, issues and questions arise as the research progresses, and often, many of what prove to be the critical issues were not even included on the initial list of questions” (Real World Evaluation, 2012, p. 55). These option substitutions do not in all cases result in cost savings; be sure to work out the details of your evaluation carefully to ensure a good option choice. For example, reducing data collection costs will not necessarily reduce analysis costs.
Advice for USING this option (tips and traps)
The validity of findings, and possibly the quantity and type of information, will be affected by these cost reduction strategies. Once you’ve adjusted the design to reduce the costs of the evaluation, return to the evaluation’s original purpose and ask again: will the evaluation be worth it? Will the new evaluation design meet the original purpose?
Be sure to triangulate options and sources to ensure accuracy when collecting limited amounts of data.
- Designing quality impact evaluations under budget, time and data constraints - Presentation on Poverty Reduction and Economic Management Network. Bamberger M, (2005)
Bamberger, M., Rugh, J., & Mabry, L. (2012). Chapter 3: Not Enough Money: Addressing Budget Constraints in RealWorld evaluation: working under budget, time, data and political constraints. In RealWorld Evaluation Working Under Budget,Time, Data and Political Constraints. (2nd ed.).Thousand Oaks, CA: Sage, (pp. 40-59) via http://www.realworldevaluation.org
Marynowski S. Denny C. & Colverson P. 2006 Best Practices Guide to Program Evaluation For Aquatic Educators Pandion Systems, Inc. Gainesville, Florida (p. 34). www.pandionsystems.com
Various summaries, presentations and other materials from RealWorld Evaluation are available in English, Spanish, French, Russian and Portuguese via http://www.realworldevaluation.org/RealWorld_Evaluation_resour.html