After the release of a new study I co-authored for the Thomas B. Fordham Institute, Social Studies Instruction and Reading Comprehension: Evidence from the Early Childhood Longitudinal Study, Tim Shanahan, a scholar and well-known commentator on literacy education, wrote a long blog post criticizing it.
I don’t know Shanahan personally, but my Fordham colleagues and I were surprised by the post, which repeatedly mischaracterized the study, made a number of odd critiques, and contained rambling sentences and perplexing claims that left us questioning our own literacy. We wrestled with how to respond. After all, Shanahan says at one point that “I fully agree with their conclusions,” and while he seems most ruffled by our idea that the two-hour block of daily English language arts (ELA) time might be pared back a bit to make room for more social studies, he at one point admits that much ELA instruction consists of useless activities that “often take up one-third to two-thirds of the instructional time.” Since he agrees with our conclusions and our (to him) most controversial point, it occurred to us that maybe we should just let the rest of it go.
But although he claims to “fully agree” with us, he also seems to think that our report is the kind of junk research that drives rigorous scholars to their fainting couches, and that only “flat earthers” would embrace it. In his attempt to convince his readers of this thesis, he repeatedly mischaracterizes our work, or even lies about what we said.
In a particularly egregious case, the post states that we “claim that cultural knowledge only comes from social studies and science classes.” This phrasing makes us sound stupid, since it is something no halfway sentient human would argue. Although the reader can reserve judgement on whether we ourselves are halfway—or even wholly—sentient, I can assure you that we never say anything like that in the report. (The report does say that “subjects such as social studies, science, and art” may be especially “content-rich,” and I continue to believe this is at the least quite plausible.)
For the moment, there isn’t time to debunk every one of Shanahan’s claims, expose all his mischaracterizations, or follow him down every rabbit hole, so instead let’s turn to his more substantive critiques:
- That we identified a correlation between social studies instructional time and reading improvement but not between ELA instructional time and reading improvement because of something to do with relative variance.
- That we have not investigated all of the intermediate mechanisms in the potential causal chain.
- That we are confusing correlation with causation.
If you are not a methodology nerd or some kind of math masochist (mathochist?), I’d advise you to skip the rest of this post and just read our report. But if Shanahan’s blog post left you feeling like we might be sloppy, crazy, lazy, or dumb, feel free to read on.
Variance
Claim: “Social studies and science get less attention, so they lead to a lot more variation in teacher practice, which increases the possibility of finding a positive correlation with achievement.”
There are two parts to this claim. The first is that the positive correlation between social studies instructional time and reading progress may be spurious. This is a half-truth. Although he says we’re more likely to find a positive correlation with learning, a spurious relationship is, in fact, just as likely to be negative as positive.
So is the relationship between social studies time and improvement in literacy spurious?
Standard statistical methods (OLS, in this case) provide not just a point estimate, but also a standard error and confidence interval, which indicate the precision with which the estimate is made. The standard interpretation of our finding is that we are more than 95 percent certain that the relationship is not due to chance. (That doesn’t mean we’re 95 percent sure that there is a direct causal relationship between the variables—see below—just that we’re not observing the relationship due to statistical noise.) This 95 percent threshold is completely arbitrary, but is the industry standard (although a lower 90 percent is common in social research as well). Not only do we find a direct, bivariate relationship between social studies time and improvement in literacy at this 95 percent threshold, but the relationship (and statistical significance) holds as we add a variety of control variables that could explain an observed spurious correlation. In the report, we call it “modest,” but readers can determine for themselves how strong of evidence of a connection that represents.
The second, related concern here is to explain that it is “peculiar that the amount of ELA time didn’t correlate with reading achievement.” The implicit claim is that we did not identify a relationship between ELA instructional time and reading improvement because there is less variance in ELA instructional time variable than in some of the other variables.
My co-author and I conducted some analyses to see if there was any possibility that this could be a problem and found no support for this concern. Since in the next breath he breezily admits that “there was certainly enough variation in the amount of ELA teaching to correlate with reading achievement,” we’ll consider that case closed.
As our report says, ELA instructional time doesn’t generally correlate with reading improvement, and social studies instructional time does. As Shanahan himself writes, perhaps that’s because many of the activities in a typical ELA block are low-value.
Mechanisms
Claim: “for this theory to be supported, you’d need to show that the social studies teaching increased student knowledge about social studies, that the students simultaneously improved in reading, and that this social studies knowledge was instrumental in the reading improvements.”
My reaction to this was: Why stop there? Theoretically, we could also classify types of social studies instruction, as well as instruction in other subjects. We could further refine those classifications. We could compare the curricula of different schools. We could investigate the frequency of vocabulary words used in different types of instruction and in different subjects. And so on.
There are probably infinite intermediate steps that no doubt would be interesting to investigate and would help us better understand how various practices relate to different types of learning growth. (We hope future researchers explore some of these!)
Social researchers almost never have complete information about every mechanism and intermediate step in the causal process. That is certainly the case for our study, which relies on data collected through the federal Early Childhood Longitudinal Study (ECLS-K). The dataset is very rich—complete with regular surveys of administrators, teachers, and parents; assessments of students in grades K through 5; and tons of information about background variables like race, gender, age, socioeconomic status, type of neighborhood, etc. Still, even with all of this information—which cost the federal government millions of dollars to collect—we don’t know everything. When it comes to instruction in the various subjects, all we know is how much time teachers report spending on each one.
So, like all scholars, we “black box” some intermediate steps while trying to imagine—and caveat—any potential threats to the validity of the causal relationship we’re investigating. For example, it’s true that we can’t prove with the extant data that “social studies teaching increased student knowledge about social studies,” Shanahan’s first complaint here. But it’s certainly far from implausible, is it not? If Shanahan or anyone else thinks that there is a step along the way that sounds backwards or seems improbable, I’m happy to consider it and determine whether we ought to adjust our analysis or add another caveat. But it is simply untrue that researchers must trace and prove every step in a causal process in order for an idea to be “supported.”
Causality
Claim: Fordham sent “correlational data to do a causal study’s job.”
Although everyone learns by their freshman year of college that correlation does not equal causation, we must not forget that correlation is, of course, often our best evidence of causation.
Do you think smoking causes cancer? Nowadays, we can point to studies using plausibly exogenous variation in smoking to establish a causal link. But all of the studies from the 1950s and 1960s that “proved” the connection between smoking and bad health outcomes were based on observational data. The tobacco companies preferred Shanahan’s method of throwing a bunch of arguments against the research and hoping that some of them would stick, but observational evidence accumulated to the point that the government issued health warnings and public opinion began to shift. (A similar process has unfolded for seatbelts, climate change, gun safety, etc.)
In other words, although we all know a correlation alone doesn’t prove causation, in the absence of confounding factors or reverse causality, it is often good evidence of causation. Our analytic approach, which resembles a difference-in-difference analysis, is designed to isolate the potential impact of different uses of instructional time, but the last paragraph of the report’s two-page methodology section issues this caveat:
Because the amount of classroom time devoted to different subjects is not randomly assigned and may correlate with other factors for which we are unable to control—such as teacher quality—it is possible that one or more of these factors is driving a correlation between class time usage and student learning. Still, a finding that different use of class time is correlated with greater improvement in reading after controlling for various potential confounders could imply a causal effect or, at the very least, suggest how better teachers employ instructional time.
The ECLS-K is a good observational data source, with both “pre-“ and “post-treatment” data and rich data to use as controls, but it is certainly true that a confounding variable could be driving the correlation between social studies instructional time and reading improvement, and we’ve always been transparent about that possibility.
—
Based on nationally representative data, our new report suggests that exposing students to more content-rich subjects like social studies will lead them to be better readers. Intuitive theories about the connections between knowledge and literacy and mounting empirical evidence point to the idea that students will become better readers faster if they focus on acquiring more knowledge of the world around them. This study, along with the broader research context, leads me to believe that carving out at least forty-five quality minutes a day for social studies should be a top priority for literacy advocates, even if some of it comes at the expense of the ELA busywork time Shanahan himself admits doesn’t do any good anyway.
