As Supreme Court justices weigh the constitutionality of Cleveland's voucher program in the next few months, their assessment of the benefits of school choice is apt to influence their decision. If there is evidence that voucher programs offer sound educational opportunities to poor children and prompt the reform of low-performing urban public schools, the "brethren" are more likely to find ways to uphold such programs' constitutionality.
Mindful that the research evidence is likely to carry weight in this important case, many organizations have sought to spin it to suit them. The main strategy of those hostile to vouchers has been to describe the evidence as "mixed" or "inconclusive" at best. Saying that research findings are inconclusive has a reasonable, moderate tone that appeals to journalists who seek balance and tend toward skepticism, even when such balance and skepticism are unwarranted.
In the case of voucher research, those labels are indeed unwarranted, though it's not hard to make it look otherwise. To turn strong and consistently positive results on behalf of vouchers into mixed and inconclusive ones, all that is needed is to apply an unreasonable high standard - perfection - to the voucher research.
One large example of this is "Rhetoric Versus Reality: What We Know and What We Need to Know About Vouchers and Charter Schools," a report issued by the Rand Corporation last month. "For most of the key questions," the Rand authors assert, "direct evaluations of vouchers and charter schools have not yet provided clear answers, and the list of unknowns remains substantially longer than the list of knowns." They do their best to cast doubt on existing studies of the effects of vouchers by using an approach most commonly seen in graduate school, where budding researchers are trained to critique extant scholarship harshly so as to hone their intellects, burnish their methodological sophistication and spot complex questions to explore in their dissertations. In the real world, however, policymakers cannot afford the graduate student luxury of critiquing ad nauseam. Since they must choose among options presented by the real world, based on the information that is available, they require a practical standard that enables them with reasonable confidence to compare various policy alternatives with the status quo. They know that more evidence will continue to roll in, policies will continue to evolve and programs will continue to be refined, replaced and renewed.
When compared with research used to support other education policies, the evidence supporting school choice is remarkably strong. Five random-assignment studies have investigated whether vouchers provide academic benefits to students who use them. They examined the impact of vouchers on children in Charlotte, Dayton, Milwaukee, New York, and Washington, D.C. Random assignment is the gold standard of research designs and program evaluations. Because chance distinguishes who is in otherwise identical treatment and control groups, this approach minimizes the risk that outcomes are caused by unobserved characteristics of students and families - or other events in their communities - rather than by the policy being studied.
All five of these studies showed that students using vouchers experience significant academic benefits. These consistently positive findings have held up under analysis and re-analysis by sundry researchers from Harvard, Princeton, the University of Wisconsin, Georgetown, Mathematica, and the Manhattan Institute. I know of only one other education policy that has been the subject of even one major random assignment study: the class-size reduction experiment in Tennessee. Most education policies are adopted without any experimental evidence, gold standard or less, concerning their effectiveness. (For example, the recently renewed federal Title I program, after 35 years, can muster no such evidence concerning its effectiveness in boosting the achievement of disadvantaged youngsters - its ostensible purpose.)
While the evidence that vouchers benefit youngsters who use them is strong, existing research has important limitations. As the Rand report correctly notes, voucher-induced learning gains are largely restricted to African-American students. Of course, this is partly due to the fact that the vast majority of students in the five voucher experiments conducted so far have been black. To learn more about possible voucher benefits for students from other backgrounds, we would need larger samples of students from these groups. Instead of hailing the finding that vouchers offer significant benefits to African-American students, however, Rand focuses on the "unknowns" for other children. Some people dwell on emptiness in glasses that are even three-quarters full.
As to whether voucher programs can have a positive impact on public schools in the district, the Rand report is even gloomier: "Whether the introduction of vouchers/charters will help or harm the achievement of students who stay in conventional public schools remains for the moment entirely unknown." The authors can say this because they excluded from their conclusion all studies that are not direct examinations of the effects of voucher programs on local public schools. For example, they give no weight to research by Harvard economist Caroline Minter Hoxby that shows that public schools demonstrate higher academic performance when they face additional competition (because they are located in metropolitan areas with more school districts). They ignore my own work with the Education Freedom Index that finds that states with a broader range of choices offered to parents have higher test scores. In short, they spurn all analyses of the current education marketplace that find that greater competition improves public school performance. This is akin to excluding animal studies when trying to determine if a chemical causes cancer in humans - or whether a new treatment might help cancer patients.
The Rand researchers also missed another important Hoxby study that directly examines the effect of new school choice programs on public school performance. She probed charter-school programs in Arizona and Michigan as well as the Milwaukee voucher program and found that the more exposed public schools were to these sources of competition, the greater their academic improvement. This important study, published in the Winter 2001 issue of Education Next, became available on the web in mid-2001, yet Rand's analysts missed it in their "exhaustive review of the existing literature on vouchers and charter schools."
On the collateral question of how school choice affects civic values, they again arrived at a negative conclusion only by excluding significant studies and relevant analyses. In this case, they missed a significant study by Georgetown professor Patrick Wolf and others that found that recipients of a voucher in the D.C. experiment were significantly more likely to be tolerant of the political activities of disliked groups (which is the conventional measure of political tolerance in social science research) than were the public school students who applied for a voucher but were denied one by lottery. Note that these differences were produced after just two years in different schools.
Such gloomy assessments of voucher research are possible despite relatively strong evidence to the contrary because the standards to which critics seek to hold voucher research are different from those they apply to other policies and programs. That's the essence of a "double-standard." Of course there's much still to be learned about the effects of school choice. But unless we try voucher programs on a larger scale and continue them for a longer period, we'll never be able to resolve today's uncertainties or gather more definitive evidence. So long as the existing research is assessed using an overly strict standard, however, policymakers may wrongly be discouraged from trying bigger and bolder programs, scholars may be discouraged from gathering further evidence and the Supreme Court may even be discouraged from allowing today's small programs to continue at all.
Jay P. Greene is Senior Fellow at the Manhattan Institute for Policy Research. The opinions expressed here are his own
