This paper, written by Katya Fels Smyth and Lisbeth B. Schorr, examines the use of experimental–design study as the standard for determining of invested dollars in programs are generating the desired results. They are argue that the assumption that everything can be judged using the same methodological approach can have severe negative consequences for stakeholders. The authors go on to outline a set of starting points for rethinking evaluation in order to provide better accountability without reducing impact.
"In assessing the success of efforts to improve outcomes for vulnerable populations, experimental methods must not be the sole arbiter of effectiveness. We pay too high a price when we give credence only to evidence that provides absolute assurance of change in a particular domain, for that threatens to skew our understanding of what constitutes a good intervention that changes lives, not one piece of a life.
We examine how experimental methods are an especially poor fit with the efforts that could help the most vulnerable populations. People who face barriers that interact and occur in clusters must be seen in their real-world contexts, taking into account their challenges and strengths, their relationships and communities. Only then are we likely to be able to respond effectively. Our evaluation methods must be modified to embrace this complexity, not simply to control for it as nuisance variables."
- What we have learned about what works
- An emphasis on relationships and trust
- An orientation toward working in partnership with program participants
- Significant front-line flexibility within established quality standards
- A deep understanding of the importance of the larger environment
- “Proof” is not enough: limitations of the experimental method
- Much is at stake
- Broadening the evaluation paradigm: a more inclusive approach
- Putting it all together
Smyth, K. F., and Schorr, L. B. (2009). A Lot to Lose: A Call to Rethink What Constitutes “Evidence” in Finding Social Interventions That Work, John F. Kennedy School of Government, Harvard University. Retrieved from: http://www.hks.harvard.edu/ocpa/pdf/A%20Lot%20to%20Lose%20final.pdf