A recent study looks at the impact of Indiana’s Choice Scholarship Program (ICSP) on school proficiency rates in math and reading at traditional public schools. Its finding that public school proficiency rates on state tests decreased in the long term after ICSP was introduced seems to suggest that the program has a negative long-term impact on public school students. But its reliance on school-level proficiency rates as opposed to student-level data means we can draw few meaningful conclusions about ICSP’s real impact.
The scholarship program, which serves low- and (more recently) middle-income families, is one of the most expansive school voucher programs in the U.S. today. Nearly eighty percent of Indiana students are eligible for the vouchers, although fewer than 4 percent currently make use of them. Because the money follows the student in Indiana, public schools have an incentive to improve their operations to compete with ICSP.
The study has the trappings of a rigorous regression analysis, including controls for student demographics and a large sample size (1268 schools in 283 districts), leading to some impressively-precise-sounding results. It found that a 1-percentage-point increase in a school district’s participation rate in ICSP resulted in a 0.922-percentage-point decrease in a public school’s math proficiency rate and a 0.840-percentage-point decrease in that school’s reading proficiency rate.
At first, one might conclude that the program harms student outcomes, but this is a statistical illusion. To understand ICSP’s impact on public school students, student-level data are needed. School-level data on proficiency rates are not enough. Suppose, for example, that students using the program in a school district are almost all achieving proficiency on state tests. If a 1-percentage-point increase in ICSP participation from such students results in a less than 1-percentage-point decrease in proficiency rates among the other students, then ICSP is actually improving student outcomes.
Even if some of the students using the program are not proficient, the results could be misleading. Imagine a population of one hundred students. Suppose that, before ICSP existed, all of these students would have attended the same public school, and that fifty of those students would have achieved proficiency at that school. That’s a proficiency rate of 50 percent. With the ICSP in place, however, twenty of these students decide to use it to attend other schools, sixteen of whom would have achieved proficiency at the public school. At the same time, the public school makes some improvements so that four students who would have not have been proficient under the old system now achieve proficiency. Suppose also that all students who would have achieved proficiency at the public school without the scholarship program also achieve proficiency at the public school with ICSP. Overall, the effect on students was positive, since four more are now proficient. But our public school now has thirty-eight students who are proficient, and forty-eight students who are not proficient, a proficiency rate of 44 percent!
In other words, this study does not reveal much about the ICSP’s competitive effects on public schools because proficiency rates are a flawed measure of school performance. That’s unfortunate because studies of the long-term impact from school voucher policies are important.
SOURCE: Yusuf Canbolat, “The Long-Term Effect of Competition on Public School Achievement: Evidence from the Indiana Choice Scholarship Program,” Education Policy Analysis Archives (July 2021).
