How population size affects PISA results

Myriad articles have pointed out that the U.S. has had very average mean scores on the recent PISA exam and that our top performers were mediocre compared to those in other countries. And while that’s true, it doesn’t tell the whole story: Size matters.

China and India didn’t participate in PISA as countries. China only submitted tests from select municipalities, and India excused itself entirely. This means that among the countries and regions that took PISA in 2012, the U.S. is the most populous—by a big margin. Below is a list of the ten most populous countries and regions that took PISA in 2012; I’ve included each of their mean scores and where those mean scores rank amongst all sixty-five.

The United States has 33 percent more people than the next-most populous country, Indonesia (which ranked second to last). As far as scores go, Japan is the outlier. But Japan also has less than half the population of the United States. And among countries with more than 100 million people, the average rank is forty-two out of sixty-five countries.

Looking at this from a different angle, here’s a list of the countries that hold the top ten mean math scores.

The average population size among these countries is 26 million, about the size of the New York metropolitan area. And, again, Japan is a huge outlier. If you ignore Japan, the average population size is fifteen million, or roughly the size of the Los Angeles metropolitan area.

Why do large countries tend to score lower? One possibility is that larger populations are harder to manage. They require more power delegation; the U.S. does this by splitting power between the municipal, state, and federal governments. Smaller countries, on the other hand, have a much easier time with this. Change is less difficult. Finland, for example, became an educational phenomenon over the last decade (although it dropped across the board this year), but the country has a population equal to that of Minnesota. The United States has forty-nine other states and over 310 million more people, which makes us roughly fifty-nine times the size of Finland. Solving our problems and better educating our students requires something entirely unique. We’re the largest developed country by almost 200 million people. Our challenge is novel.

Furthermore, in the real world, raw numbers matter. If Macao had 10 percent of its population achieving at PISA’s highest level, it’d have 100,000 elite performers. If the U.S. had half that proportion, 5 percent, the US would have sixteen million elite performers. Sixteen million! To better illustrate this, here is a list of the top ten countries by total number of students at Level 6, PISA’s top proficiency level.*

The United States has the third highest number of elite performers in the world, and it’s virtually identical to the wildly high-flying PISA powerhouse Shanghai.

This, of course, is terrible for the U.S. on the flip side of the coin. We have a large population and a disproportionate number of low performers, and that’s a huge problem. We should also always strive to improve. But when we talk about how the U.S. is dangerously far behind top-scoring countries in producing elite performers, and before we rush to the presses with frenzied tales of apocalyptic economic catastrophe, let’s take a step back and realize that, in context, we’re still producing more of the world’s educated elite than almost any other country.

* For this table, I took the percentage of students at Level 6 and multiplied it by population, rounding that number to the nearest hundred thousand. This would be the number of elite performers the country would eventually produce if its education system continued to perform at the present level and its population size remained unchanged.