Regression Discontinuity Design (RDD) is a quasi-experimental evaluation option that measures the impact of an intervention, or treatment, by applying a treatment assignment mechanism based on a continuous eligibility index which is a variable with a continuous distribution.
For example, test scores of students can take any value between 0-100 and area of land owned can have infinite values. If treatment is assigned to those either above or below a certain “cut-off” point, RDD can be used to measure the difference in outcomes of individuals, households, or communities clustered around the defined cut-off point. This necessitates the determination of a ‘bandwidth’ around the cut-off point within which individual units are shown to be statistically comparable. If the assumptions of the option hold, the difference in outcomes between those above and below the cut-off can be attributed to the program.
School Enrollment, Selection and Test Scores
The study (see table below) analyzed the impact of a scholarship program, the Cambodia Education Sector Support Program (CESSP), on school enrollment, selection, and test scores using a sharp regression discontinuity design (RDD). Eligibility for the $45 scholarship depended on an independently calculated dropout risk score created using household characteristics. The data used included the composite dropout risk scores, mathematics and vocabulary test scores, and a household survey. The cut-off score varied by school size, and the estimates were weighted averages. The sample used in the analysis consisted of children within a ten-point bandwidth range around the cut-off score. The evaluation found that school enrollment and attendance increased 20-25% due to the scholarships. Years of schooling increased by 0.21 years, while annual educational expenses paid by the family rose by $9. The actual program impact was likely to be higher since recipients were, on average, poorer than non-recipients. No significant impacts on learning outcomes were found. However, recipients had better knowledge of HIV/AIDS, and the program had a positive effect on the recipients’ mental health. (Filmer & Schady, 2009)
- Treatment is assigned on the basis of an observable variable or index
- There is a discontinuity in the probability of being treated at some cut-off value of the assignment index or variable
- The cut-off value of the index for treatment is arbitrary, and thus units on either side of the cut-off point are identical, on average, except for the presence of the treatment
- The same cutoff is not used for determining eligibility for multiple treatments (e.g. households below a certain poverty line have access to multiple government interventions)
Program effects on school attendance
- Used in instances where treatment is assigned according to a discrete cut-off point with all eligible receiving the treatment and all ineligible not receiving it. A simple comparison of means is needed to calculate the effect of the treatment.
- Used in instances where some eligible fail to receive treatment and/or ineligible receive it (e.g. due to self-selection or administrative overrides). The fuzzy version is more often utilized in practice.
- To calculate the effect of the treatment: a) Regress observable characteristics on outcomes to obtain the ‘outcome discontinuity', and b) Regress observable characteristics on the treatment indicator to obtain ‘treatment discontinuity.’ Then, Treatment effect = outcome discontinuity / treatment discontinuity
Advice for CHOOSING this option (tips and traps)
RDD can be used in situations where:
- A continuous eligibility index (e.g. test scores, age etc.) is used to rank the population and
- A clearly defined cut-off point that determines eligibility for treatment is part of the program design and
- The same cut-off score has not been used in determining eligibility for other treatments.
- Be aware that RD provides limited external validity as results are only generalizable around the cut-off - provision of a service might make more or less difference to people who are further away from the cut-off point.
Advice for USING this option (tips and traps)
- Test for the existence of a discontinuity in the probability of receiving treatment (Nichols, 2009)
- Test for the comparability of units within the bandwidth using ‘leave-one-out’ cross validation test (Ludwig and Miller 2007, Imbens and Lemieux 2008) or asymptotic theory (Imbens and Kalyanaraman 2009).
- Depending on the type of treatment there may be ‘cross-contamination’ between those eligible for the intervention and those who are not.
Handbook on Impact Evaluation: Quantitative Options and Practices: Chapter 7 and 16 of this book, from the World Bank, give detailed descriptions and examples of the use of this option in impact evaluation.
- Strengthening The Regression Discontinuity Design Using Additional Design Elements: This paper, written by Coady Wing and Thomas D. Cook, demonstrates that through the use of an untreated comparison to the basic regression discontinuity design, it is possible to overcome three of the main problems associated with it compared to randomized control trials (RCT)
Buddelmeyer, H., & Skoufias, E. World Bank, (2004). An evaluation of the performance of regression discontinuity design on PROGRESA (World Bank Policy Research Working Paper 3386). Retrieved from website: http://www-wds.worldbank.org/servlet/WDSContentServer/WDSP/IB/2004/09/09/000009486_20040909125120/additional/103503322_20041117144006.pdf
Campbell, D. T. (1971). Reforms as experiments. Urban Affairs Review, 7, 133-171. Retrieved from http://uar.sagepub.com/content/7/2/133.full.pdf html
Filmer, D., & Schady, N. The World Bank, Human Development and Public Services Team. (2009).School enrollment, selection and test scores (Policy Research Working Paper 4998). Retrieved from website: http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2009/07/17/000158349_20090717130545/Rendered/PDF/WPS4998.pdf
Hahn, J., Todd, P., & Van der Klaauw, W. (2001). Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica, 69(1), 201-209. Retrieved from https://www.betterevaluation.org/sites/default/files/hahn%20et%20al%2000.pdf
Imbens, G., & Lemieux, T. (2007). Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142(2), 615-635. Retrieved from http://faculty.arts.ubc.ca/tlemieux/papers/designs.pdf.
Khandker, S. R., Koolwal, G. B., & Samad, H. A. The International Bank for Reconstruction and Development / The World Bank, (2010).Handbook on impact evaluation quantitative options and practices (52099). Retrieved from website: http://www.esf-agentschap.be/uploadedFiles/Voor_ESF_promotoren/Zelfevaluatie_ESF-project/statistiek%20voor%20quasi%20experimenten.pdf
Nichols, A. STATA, (2009). Causal inference with observational data regression discontinuity and related options in STATA. Retrieved from website: http://www.stata.com/meeting/germany09/nichols.pdf