Understanding Variation in Treatment Effects in Education Impact Evaluations: An Overview of Quantitative Methods

This report, written by Peter Z. Schochet, Mike Puma and John Deke for the Institute of Education Sciences (IES), aims to create a summary of the research literature on quantitative methods for assessing the differences that educational interventions have on instructional practices and student learning that exist across students, educators, and schools and outline some guidance on the use and interpretation of these methods. 

Excerpt

"A key purpose of rigorous evaluations of education programs and interventions is to inform policy choices. Typically, such assessments focus on the overall or average treatment effect of the intervention on key outcomes. However, there are also important program and policy questions that pertain to variation in treatment effects across subgroups of study participants, as defined by their baseline characteristics, local area contexts, and program experiences. Variation in effects has important implications for educational practice—and for facilitating the most efficient use of limited resources—by informing decisions about how to best target specific interventions and suggesting ways to improve the design or implementation of the tested interventions. Understanding variation in effects is also critical to assessing how findings from a particular study or set of studies may be generalized to broader educational environments."

Contents

  • Background: What are potential sources of variation in treatment effects? 4
  • Pre-intervention influences 4
  • The magnitude and nature of the treatment-control contrast 5
  • Topic 1: What are treatment effects for subgroups defined by baseline characteristics of students, teachers, and sites? 6
  • Topic 2: To whom do the results of this evaluation generalize? 13
  • Topic 3: What mediating factors account for treatment effects on longer-term outcomes? 18
  • Topic 4: What are treatment effects for subgroups defined by individuals’ post-baseline experiences 27
  • Topic 5: Do treatment effects vary along the distribution of an outcome measure, such as a student achievement test score? 30
  • Topic 6: What impact estimation methods are appropriate when treatment effects are heterogeneous? 35 

Sources

Peter Z. Schochet, Mike Puma and John Deke (2014). Understanding Variation in Treatment Effects in Education Impact Evaluations: An Overview of Quantitative Methods, Institute of Education Sciences (IES). Retrieved from: http://ies.ed.gov/ncee/pubs/20144017/pdf/20144017.pdf