In a recent AEI meta-analysis of school choice attainment literature, Michael McShane, Patrick Wolf, and Collin Hitt use thirty-nine impact estimates from studies of more than twenty school choice programs to argue that standardized-test impacts are too unreliable to serve as the “exclusive or primary metric on which to evaluate school choice programs.” In their words:
Programs that produced no measurable positive impacts on achievement have frequently produced positive impacts on attainment. And on the other hand, null effects on high school graduation and college attendance have been reported from programs that produced substantial test score gains. Across these studies, achievement impact estimates appear to be almost entirely uncorrelated with attainment impacts.
Are they right about that last part? As avid Fordham readers know, my colleague Mike Petrilli has already criticized the authors’ methodology and conclusions at length. But for those of you who don’t have time for Mike’s six-part mini-series, here is my abbreviated critique.
First, for a study’s achievement and attainment estimates to “match” under the authors’ methodology, both the sign and their statistical significance of those estimates must be the same. So, for example, if one estimate is positive and the other is negative, they are considered mismatched, even if both estimates are tiny and statistically insignificant. Similarly, estimates are considered mismatched if one is significant and the other is insignificant, even if both estimates are positive (or negative) and similar in magnitude.
If you know any statistics—and the authors do—than you can spot the problem here. As they admit, a fairer approach would be to “simply examine the correlation between program effect sizes on each outcome.” But their actual approach effectively suppresses this correlation. For example, according to McShane, Wolf, and Hitt, English language arts achievement and high school graduation only match in thirteen of thirty-four instances—leaving twenty-one supposed mismatches. Yet according to the authors, only one of those studies found significant effects on achievement and attainment that pointed in opposite directions. (And even in this case, it sure looks like they got their facts wrong.) Meanwhile, seven studies found significant effects that pointed in the same direction. (I guess that’s what happens when two outcomes are almost uncorrelated.)
Second, the authors use their “findings” for program impacts to argue that all test-based accountability is flawed. But of course this is a massive non sequitur. As Mike noted in his second column, there is far more variation in the performance of individual schools than there is in the average performance of school choice programs. So even if achievement and attainment impacts weren’t strongly correlated at the program level (which they might be), there would still be a case for closing schools with extremely low test scores (which is what accountability hawks are actually suggesting).
On a happier note, although the authors’ analysis only includes studies that estimate both achievement and attainment impacts—thus excluding a large number of studies that only estimate the former—the news for school choice is overwhelmingly good. For example, of the thirty-four estimates in the authors’ sample that considered ELA achievement, eleven found a significant positive effect, while just three found a significant negative effect. Similarly, eleven estimates found a significant positive effect for math, while just one found a significant negative effect. And sixteen found a significant positive effect on high school graduation, while just two found a significant negative effect. Finally, nine of nineteen studies found a positive impact on college attendance, and three of eleven found a positive effect on four-year college completion. Yet no study found a significant negative effect on postsecondary attendance or completion. (Think about that for a second.)
In short, the study is full of good news for school choice advocates, and a careful reading actually strengthens the case for taking the achievement impacts of these programs seriously. (For example, ELA impacts seem to be a better predictor of long-term gains than high school graduation rates.) So it’s a shame that this whole conversation has been sidetracked by a shoal of red herrings.
Traditionally, when social scientists have chosen methods that fit their preferred conclusions, they have done so in secret. This report has the debatable virtue of transparency.
SOURCE: Michael Q. McShane, Patrick J. Wolf, and Collin Hitt, “Do Impacts on Test Scores Even Matter? Lessons from Long-Run Outcomes in School Choice Research,” American Enterprise Institute (March 2018).
