The first pandemic-influenced data from the Progress in International Reading Literacy Study (PIRLS) test are in. Unsurprisingly, an initial analysis says the news is bad.
PIRLS is administered every five years to a representative sampling of fourth grade students in more than fifty countries and regions around the world (including the U.S. and other industrialized nations, as well as some less-developed countries). It provides internationally comparative data on how well children read; the trend data are meant to measure the big picture ups and downs in reading achievement over time. This report offers almost no nation-specific results—including for the United States.
The new study compares results from assessment rounds in 2001, 2006, 2011, 2016, and the most recent in 2021. The sample includes more than 1 million students from each administration, but represents more than 16 million boys and girls worldwide. Achievement is compared using traditional socio-demographic characteristics, but in the years since Covid disrupted education worldwide, the influence of school closures has become a new data point of extreme interest. To wit: Principals in PIRLS schools were asked to estimate the number of weeks during the testing year “where normal primary school operations have been affected by the COVID-19 pandemic.’’ As an additional measure of pandemic disruption, the researchers utilized UNESCO data on the length of government mandated school closures in countries and regions.
The researchers model the effect of school closures on student achievement by predicting the deviation of the most recent 2021 results from an estimated linear trend in reading achievement using data from the four previous testing rounds included in the analysis. They estimate each country or region’s linear trend separately—including country-level fixed effects to control for unobserved characteristics—and report the difference between the estimated and the actual outcomes. Overall, reading achievement declined by 15 points on average across countries, or 18 percent of a standard deviation (SD). Since some countries delayed test administration due to the pandemic, the researchers also adjusted their analysis to account for students of differing ages and grade levels. This effort finds an even larger decline: 27 points, or 33 percent of an SD overall, which the researchers point out is roughly equivalent to a year’s worth of learning loss. Some countries, such as India at 73 points, lost even more.
The longer schools stayed closed, the worse their students performed on the PIRLS test. Countries with no school closures per UNESCO data (such as Australia and Sweden) achieved results in 2021 that accord with predicted outcomes. That is, they saw no decline in achievement based on previous trends. With each week of full or partial school closures, though, the results decline by 0.8 points from predictions. Thus, ten weeks of full or partial closures resulted in an 8-point decline (or 9 percent of an SD), twenty-five weeks resulted in a 20-point decline (24 percent of an SD), and fifty weeks resulted in a nearly 40-point decline (47 percent of an SD). To highlight India once again, their schools were partially or fully closed for 93 weeks, and its students registered one of the largest learning losses per this international test.
There was no difference in the findings for boys versus girls, but lower-achieving students (here defined as the first decile of achievement) experienced much larger losses than their higher-achieving peers. For example, in countries with ten weeks of full or partial closures, high achievers experienced no loss, while the lowest performing students lost 12 points. But in countries with fifty weeks of full or partial closures, the best students lost 28 points, while their low-achieving peers lost 37 points. Interestingly, there was barely any difference in outcomes for countries which postponed their PIRLS administration compared to those which did not postpone. This suggests that, for up to a year, countries’ efforts to mitigate the losses experienced during the pandemic were minimally effective.
Interested readers can peruse supplementary data, which is broken down by country/region, indicating that there is considerable heterogeneity across them, especially in terms of student achievement in the face of school disruptions. But there are no direct comparisons between countries provided by these researchers. The new report, unfortunately, leaves it at the overall finding for this one international test of reading achievement: A bleak picture of global learning loss directly tied to the length of school closures.
SOURCE: Maciej Jakubowskia, Tomasz Gajderowicza, and Harry Anthony Patrinos, “Global learning loss in student achievement: First estimates using comparable reading scores,” Economics Letters (August 2023).
