Evaluation of Ohio’s EdChoice Scholarship Program: Foreword

Shortly after Ohio lawmakers enacted a new voucher program in 2005, the state budget office wrote in its fiscal analysis, “The Educational Choice Scholarships are not only intended to offer another route for student success, but also to impel the administration and teaching staff of a failing school building to improve upon their students’ academic performance.” As economist Milton Friedman had theorized decades earlier, Ohio legislators believed that increased choice and competition would boost education outcomes across the board. “Competition” in the words of Stanford’s Caroline Hoxby, “would be the proverbial rising tide that lifts all boats.”

Today, the EdChoice program provides publicly funded vouchers (or “scholarships”) to more than eighteen thousand Buckeye students, youngsters previously assigned to some of the state’s lowest-performing schools, located primarily in low-income urban communities.[1] That much is known. Yet remarkably little else is known about the program. Which children are using EdChoice when given the opportunity? Is the initiative faithfully working as its founders intended? Are participating students blossoming academically in their private schools of choice? Does the increased competition associated with EdChoice lead to improvements in the public schools that these kids left?

The present study utilizes longitudinal student data from 2003–04 to 2012–13 to answer these important questions. Specifically, the analysis utilizes the results from state tests—which all EdChoice students are required to take—to examine the vouchers’ effects on two groups of pupils. First, the study inspects the scores of public school students who were eligible for vouchers—but did not take one—in order to gauge the competitive effects of EdChoice (i.e., its impact on traditional public school students and their schools). Second, it examines the academic impact of EdChoice on those students who actually use the vouchers to attend private schools.

This is the first study of EdChoice that uses individual student-level data, allowing for a rigorous evaluation of the program’s effectiveness. (Earlier analyses by Matthew Carr and Greg Forster used school-level data to explore its competitive impact.) To lead the research, we tapped Dr. David Figlio of Northwestern University, a distinguished economist who has carried out examinations of Florida’s tax credit scholarship program. He has also written extensively on school accountability, teacher quality, and competition. Given his experience, Dr. Figlio is exceptionally qualified to lead a careful, independent evaluation of Ohio’s EdChoice program.

In this report, he sets forth three main findings:

• While the students who participate in EdChoice—the pupils who actually use a voucher to attend private schools—are primarily low-income and minority children, they are relatively less disadvantaged than other voucher-eligible students. Figlio reports that more than three in four participants are economically disadvantaged, and three in five are black or Hispanic. Viewed in relation to Ohio’s public school population as a whole, students in EdChoice are highly disadvantaged—not surprising, given eligibility rules that require participants to have attended a low-achieving public school. But relative to students who are eligible for vouchers but choose not to use them, the participants in EdChoice are somewhat higher-achieving and somewhat less economically disadvantaged. This finding may be, in part, an artifact of the program’s basic design: It allows private schools to retain control over admissions, and a child must gain admission into a private school before he or she can apply for a voucher. This multi-step process might be more easily navigated by relatively more advantaged families; their children might also be more likely to meet the private schools’ admissions requirements.
   
• EdChoice improved the achievement of the public school students who were eligible for the voucher but did not use it. When examining the test results of pupils attending public schools just above and below the eligibility threshold, the analysis finds that achievement in math and reading rose modestly as a result of voucher competition. (The analysis leverages the state’s voucher eligibility rules to isolate voucher competition from other potential competitive effects, such as charter schools.) In other words, the voucher program has worked as intended when it comes to competitive effects. Importantly, this finding helps to address the concern that such programs may hurt students who remain in their public schools, either as a result of funds lost by those schools or the exodus of higher-performing peers. Quite the opposite has occurred in the case of EdChoice: Achievement improved when the voucher program was introduced and public schools faced stiffer competition (and the risk of losing their own students).
   
• The students who use vouchers to attend private schools have fared worse academically compared to their closely matched peers attending public schools. The study finds negative effects that are greater in math than in English language arts. Such impacts also appear to persist over time, suggesting that the results are not driven simply by the setbacks that typically accompany any change of school.

Let us acknowledge that we did not expect—or, frankly, wish—to see these negative effects for voucher participants; but it’s important to report honestly on what the analysis showed and at least speculate on what may be causing these results. One factor might be related to the limits of credible evaluation: while the rigor of the methodology ensured “apples-to-apples” comparisons of student achievement, Dr. Figlio was limited to studying students who attended (or had left) public schools that were just above or below the state’s cutoff for “low-performing.” By definition, this group did not include the very lowest-performing schools in the state. It’s possible that students who used a voucher to leave one of the latter schools might have improved their achievement; we simply cannot know from this study. The negative effects could also be related to different testing environments—higher stakes for public than private schools—or to curricular differences between what is taught in private schools and the content that’s assessed on state tests. Finally, although this analysis does not enable us to identify individual schools as high- or low-performing, it may be the case that some of the private schools accepting EdChoice students are themselves not performing as well as they should.

***

Taken as a whole, the results reported here for Ohio’s EdChoice program—one of the nation’s largest voucher programs—are a mixed bag. The program benefitted, albeit modestly, thousands of public-school students; yet among the somewhat small number of participants studied here, the results are negative. The study mirrors important trends that can be seen in other voucher research. The modest, positive competitive effect on public school achievement replicates findings from jurisdictions like Florida, Louisiana, and Milwaukee, findings that also offered evidence that voucher competition improved public school outcomes. These are, of course, encouraging for advocates of competition and choice. Yet this study also extends a recent (and, to us, unwelcome) trend that finds negative effects for voucher participants in large statewide programs. While earlier evaluations of privately and publicly funded scholarship programs—usually administered at the city level—found neutral-to-positive impacts on participants, newer studies of Louisiana’s and Indiana’s statewide programs have uncovered negative results, particularly in math.

There’s been much discussion about what might be behind these participant results. Is too much regulation discouraging high-quality private schools from joining the program? Are state exams failing to capture important private school contributions to student success? Do large, statewide programs lack the tools and resources to ensure quality at scale? Or are private schools simply struggling to raise achievement—especially in math—in relation to their public school counterparts? Some or all of these (or other) factors may be at work, but no one really knows for certain. More research on the effects of statewide voucher programs is obviously warranted.

Even though we don’t have all the answers, we believe that thoughtful policy makers can draw from the extant research as well as on-the-ground experience to give these programs the best chance of succeeding for more students, whether attending public or private schools. The pertinent lessons seem to us applicable both in states considering new private school choice programs and in states (like Ohio) that are seeking to improve an existing program.

First, we need to foster a healthy, competitive environment in K–12 education. A competitive jolt can awaken sleepy, lazy, or slipshod schools to clean up their act and attend more closely to the academic needs of their students. On the policy side, this means that lawmakers should continue to encourage a rich supply of school options, including not just private schools (in their many flavors, including religious and non-sectarian) but also public charter, STEM, and career and technical schools. At the same time, families can do their part by demanding more quality school choices. Competition and choice—two sides of the same coin—can incentivize all schools to work harder at meeting the needs of their pupils.

Second, policy makers should resist calls to pile more input-based regulations upon voucher-accepting private schools. Ohio’s private schools already face heavier regulation than those in many states. For example, they must adhere to state operating standards and hire state-licensed or certified teachers. Most of this was true before EdChoice came along (which makes less likely the “overregulation” explanation for disappointing participant results, at least in Ohio). Policy makers should tread lightly when adding to schools’ regulatory burdens: After all, freedom from regulation is precisely what makes private schools different and—for many—worth attending in the first place.

Third, as this study suggests, private schools likely vary when it comes to quality, and the public needs maximum transparency about this. Accordingly, state leaders should help families better understand the quality of their options by providing easy-to-compare information on the performance of voucher-accepting private schools. While Ohio already reports voucher students’ proficiency rates at the school level (subject to FERPA limitations), we know that those results are likely to be conflated with non-schooling factors like family income. They are also hard to track down. To be fair to private schools that educate disadvantaged voucher pupils, we suggest the adoption of a value-added measure—a school quality indicator that is more poverty-neutral than conventional academic proficiency rates. States (including Ohio) should make sure that these academic outcomes for voucher-accepting private schools are easily accessible to parents, perhaps in a report card-like format akin to those adopted for public schools. In Ohio, this would not add any additional testing or regulatory requirements on private schools.

Fourth, policy makers should craft simple, parent-friendly program rules. From the perspective of families, EdChoice is fairly complex, which may have influenced who participates in it. Eligibility hinges on public schools’ annual ratings from the state—which can change from year to year—and the state has no obligation to notify parents of their children’s eligibility. This means that families must bestir themselves to visit the state’s website or seek eligibility information through other channels. To ensure awareness, states should require direct notification of eligibility from the state department of education or a competent nonprofit agency. (This should also happen when eligibility is based on income.) Making matters more complicated, current EdChoice application rules require eligible students first to gain admission to a private school; then the school applies to the state for a voucher. It would be far simpler for parents if they could apply directly to the state for a voucher and then shop for the right private school. This process would not only empower parents but also give policy makers a much clearer picture of the demand for vouchers.

The present report breaks important new ground, but it is by no means the final word on EdChoice. We still have much to learn, including whether vouchers impact non-testing outcomes such as post-secondary success. We also need a deeper understanding about the quality of individual private schools. But the information set forth in the pages that follow is critically important as thoughtful policy makers consider the design and implementation of voucher programs, both in Ohio and across the nation. Programs that aim to better the lives of children must face scrutiny from independent, credible evaluators. Even when its findings are unexpected and painful, rigorous, disinterested evaluation remains the best way to prod improvements and make progress toward the program’s goals. In the case of EdChoice, the program appears to have met one of the two objectives conceived by its founders: Competition has spurred some public school improvement. The challenge ahead is to forge a stronger EdChoice program, one that can lead to widespread academic improvements for children who take their scholarships to the state’s private schools.