A new study examines the impact of New Orleans’s market-based education reforms on a wide range of academic outcomes. Overall, the authors—Douglas Harris and Matthew Larsen, of Tulane University and Lafayette College, respectively—estimate that these reforms increased English language arts and math achievement by 11–16 percentiles, in addition to boosting the high school graduation rate by 3–9 percentage points. Similarly, they estimate that the reforms boosted college entry (by 8–15 percentage points), persistence (by 4–7 percentage points), and graduation (by 3–5 percentage points).
Practically speaking, these are large impacts. For example, a 15-percentage-point increase in the college entry rate is roughly equivalent to a 67 percent increase. However, as the authors acknowledge, there is considerable uncertainty associated with their estimates.
Because of its unique circumstances, New Orleans presents an unusual number challenges for researchers. Consequently, Harris and Larsen take two approaches to analyzing the data: First, they conduct a “returnees-only analysis,” which considers only those students who returned to New Orleans after Hurricane Katrina. Second, they conduct a “cross-cohort analysis,” which allows them to consider a wider range of outcomes, but does not allow them to track the progress of individual students before and after the storm.
Notably, the returnees-only analysis yields smaller estimates than the cross-cohort analysis, suggesting that the latter may overestimate the benefits of the reforms. However, because returning students had lower pre-Katrina test-scores than non-returnees, it’s also possible that the returnees-only analysis underestimates the benefits of reform. (And of course, the trauma and disruption of the hurricane could be pushing all of the authors’ estimates down.) Hence, although it’s difficult to know which of the authors’ estimates are most credible, the evidence suggests that the returnees-only estimates are a true lower bound—meaning that students definitely made some gains following the reforms.
Perhaps the biggest threat to Harris and Larsen’s conclusions stems from “the infusion of funding that coincided with the reforms.” (A previous study documented a 13 percent increase in school spending in New Orleans relative to the comparison group.) As they note, it’s likely that some of this increase was actually an effect of the reforms. Still, insofar as this funding increase contributed to the positive effects the authors document, the implications for policy are different.
Alas, it’s exceptionally difficult to separate the effects of this funding increase from the effects of the market-based reforms Harris and Larsen focus on, though prior research does suggest that much of the improvement that New Orleans experienced during this period was the result of school closure and turnaround efforts, which is more consistent with the reform hypothesis.
In addition to that caveat, as the authors note, there are reasons to question the replicability of the New Orleans model. For example, because it was the first major city to attempt dramatic reforms, New Orleans likely had a “first-mover” advantage that allowed it to attract unusually ambitious and talented people. And conversely, the sheer dysfunction of the New Orleans school district pre-Katrina may have given reformers an unusually low bar to clear.
Still, if the most important question for policymakers is whether the reforms worked, then this study provides further evidence that the only sensible answer is ”yes.”
SOURCE: Douglas N. Harris and Matthew F. Larsen, “What Effect Did the New Orleans School Reforms Have on Student Achievement, High School Graduation, and College Outcomes?” Education Research Alliance for New Orleans (July 2018).
