A recently released report by the Council of the Great City Schools seeks to determine whether urban public schools—including charters—are succeeding in their efforts to mitigate the effects of poverty and other educational barriers.
To conduct the analysis, the council used student-level data from administrations of the National Assessment of Educational Progress (NAEP) from 2009 through 2019 in math and reading for grades four and eight. The analysis compares two mutually exclusive and non-overlapping groups: Large City Public Schools (LCPS) and All Other Schools (AOS), a category that includes both public and private schools. Both groups include charters.
In terms of demographics, the makeup of students in large city schools is substantially different than that of all other schools. The population of LCPS was more predominantly Black and Hispanic, and these schools were more likely to have higher numbers of students who were eligible for free or reduced-price lunch or were identified as English learners. In fact, students in large city schools were approximately 50 percent more likely to be poor, twice as likely to be English learners, and twice as likely to be Black or Hispanic. Large city schools also tended to have a higher percentage of students whose parents didn’t finish high school.
The council begins its analysis by comparing actual NAEP performance levels—unadjusted scale scores—for large city schools and all other schools. Given the demographic differences outlined above, large city schools unsurprisingly scored below all other schools on every NAEP administration between 2009 and 2019. However, LCPS improved their performance faster than AOS. In fact, gaps between the nation’s urban schools and all other schools narrowed by between one-third and nearly one-half during this time period, depending on grade and subject level.
Next, the council compared large city schools and all other schools using adjusted scale scores. They controlled for a laundry list of demographic variables, allowing them to statistically predict expected results and compare those to actual NAEP results. The difference between actual and predicted scores was dubbed “the district effect,” and was used to identify which urban school districts produced enough “educational torque” to mitigate poverty and other barriers. (Keep in mind that, despite the nomenclature, the analysis includes charter schools.)
The results are clearly in the favor of large city schools. Their effects were larger than expected for every NAEP administration in the study’s timeframe with the exception of eighth grade reading in 2011 and 2013. AOS did show significant gains in district effects between 2009 and 2019. But LCPS boasted district effects that, when compared to AOS, were 1.8 times greater in fourth grade reading, 5.6 times greater in eighth grade reading, 2.5 times greater in fourth grade math, and 3.6 times greater in eighth grade math. These large effects are likely what’s helped narrow the gap in unadjusted scale scores between the two groups.
The council also examined data from the Trial Urban District Assessment (TUDA), a voluntary initiative that over-samples students in participating NAEP districts to obtain district-level estimates of reading and math performance. These data allowed researchers to examine city-specific results by looking at the district effect of TUDA participants. Charter schools are included in these findings when TUDA samples incorporated them, but they were excluded for districts where charters are independent and therefore not counted in the district’s scores. The council examined the 2019 fourth grade results of twenty-seven jurisdictions and found that seventeen showed statistically significant positive effects in math and fifteen did so in reading. There were only twenty-six jurisdictions in the eighth-grade analysis (student questionnaire data wasn’t collected in Denver, so adjusted NAEP scores weren’t available), but the results were similar. In eighth grade math, fifteen locales had positive district effects compared to nine cities with those effects in reading.
Five jurisdictions registered positive effects in all four test areas: Atlanta, Boston, Hillsborough County, Miami-Dade County, and Chicago. Another nine, including Cleveland, demonstrated positives in three of the four areas. Some locales are also exhibiting impressive growth over time. In fourth grade math, for instance, there are three places that went from negative impacts in 2009 to positives in 2019: Chicago, Cleveland, and the District of Columbia.
To better understand how jurisdictions improved, the council visited six that showed substantial progress between 2009 and 2019—Boston, Chicago, Dallas, the District of Columbia, Miami-Dade County, and San Diego. These places took different approaches to reform, but shared several common features such as strong and stable leadership, accountability and collaboration, intentional support for struggling schools and students, and community investment and engagement. For instance, Dallas implemented an initiative called Accelerating Campus Excellence, which identified historically failing schools and provided them with prescriptive and data-driven instructional practices, schoolwide systems for social and emotional learning, extended learning time, and classroom upgrades. Miami-Dade, meanwhile, paired its support for struggling schools with school choice initiatives. And while the report doesn’t spend much time discussing school choice, it’s worth noting that a vibrant school choice sector is something many of these fast-improving places have in common.
Overall, these data should be encouraging for big-city reformers. It shows that urban public schools are outperforming expectations on NAEP, and that they “seem to be doing a better job than other schools at dampening the effects of poverty, English language proficiency, and other factors that often limit student outcomes.” Several TUDA districts, such as Cleveland, Chicago, and the District of Columbia, showed impressive improvement between 2009 and 2019. And though the council doesn’t explicitly mention school choice as a factor, many of the jurisdictions they identified as models for improvement have vibrant school choice sectors. Obviously, there is no silver bullet. But the most successful cities appear to have several strategies in common, and these strategies should definitely be part of any reform efforts going forward.
SOURCE: Michael Casserly, Ray Hart, Amanda Corcoran, Moses Palacios, Renata Lyons, Eric Vignola, Ricki Price-Baugh, Robin Hall, and Denise Walston, “Mirrors or Windows: How Well Do Large City Public Schools Overcome The Effects Of Poverty And Other Barriers?” Council of the Great City Schools (June 2021).
