The Education Gadfly Weekly

This is the fifth and final post in a series of commentaries leading up to the release of new NAEP results on April 10. The first post discussed the value of the NAEP; the second looked at recent national trends; the third examined state-by-state trends; and the fourth reviewed urban trends. Also see Ian Rowe’s post making the case that NAEP should track results by family structure.

When the National Assessment of Education Progress results are released on Tuesday, reporters, educators, and policy wonks will have a lot to digest. Over the past several weeks, I’ve examined recent trends at the national, state, and local levels. First let me review the highlights, then identify seven stories to watch when the new data go live.

• After big increases in the late 1990s and early 2000s, especially in math, especially in fourth grade, and especially for our lowest-performing students and disadvantaged subgroups, national trends have been mostly flat for almost a decade.
• However, the national picture hides a lot of state-by-state variation. In the past, states including North Carolina, Texas, Florida, and Massachusetts have made big improvements. More recently, the District of Columbia, Indiana, and Tennessee have posted big gains, overall and for most subgroups. On the flip side, quite a few states have lately regressed in math; Texas and Kansas saw some of the most widespread declines.
• Among large urban districts, the progress of D.C. Public Schools, Chicago, and Miami is especially impressive since 2011.

With that in mind, what should we be looking for next Tuesday?

1. Will the flat national trends continue? Given recent history, this is the safest bet. But predictable doesn’t make it any less disappointing. It will be time to start talking about the “lost decade” for educational progress in the United States. The question is why we hit a wall in the late 2000s. Was it the recession? Kirabo Jackson thinks it could have been. Was it the accountability movement running out of steam? Something else? We can speculate, but it will take rigorous scholarship to ferret this out.
2. Did the switch to tablet-based assessments have an impact on the scores? The 2017 score release was delayed by months because the experts at the National Center for Education Statistics wanted to make sure that the switch from paper-and-pencil testing to tablet-based assessments—fully implemented this round for the first time—didn’t mess up the findings. But were they able to adjust the scores appropriately, including at the state and local levels? Louisiana state superintendent John White is not so sure. If scores drop among low-income and low-performing students—the kids least likely to be comfortable with digital devices, especially in the fourth grade—that could signal that something went awry, possibly that the scores aren’t truly comparable with those that came before. And that would have a particularly large impact on states and districts serving lots of disadvantaged students—especially because NCES included a paper-and-pencil sample at the national level large enough to ensure consistent national trends but did not do the same at the state and local levels.

Another question to ask: Did the states that still use paper-and-pencil tests for their own annual assessments do worse? Most states now employ online tests for grades three through eight, with exceptions allowed for students with disabilities. But there some paper-and-pencil holdouts, and their students may be at a disadvantage on the new NAEP. Seven states give districts the option of using paper-and-pencil tests: Iowa, Louisiana (for grades three and four), New York, North Carolina, Pennsylvania, South Carolina, and Texas. Another two use paper-only assessments through grade four: Oklahoma and Tennessee. And Kentucky uses only paper tests for all grade levels. If these states in particular see declines, it could be because of the tablet-based testing format.

3. What’s the verdict on the Michelle Rhee/Kaya Henderson era in Washington, D.C.? After years of positive press and lavish praise from reformers, the District of Columbia Public Schools have faced a series of scandals in recent years. A debate is now raging about whether the remarkable progress reported there was real. The new scores will give us more clues about which view is right. Keep in mind that we should look at trends over at least four years, and especially for subgroups, given D.C.’s rapidly changing student demographics. But if this is the year that D.C.’s meteoric rise in NAEP scores ends—a rise that started before Rhee—it will surely strengthen the argument of those who think that the DCPS reforms were overhyped.
4. Is Indiana still an Ed-Reform Idol? The state’s progress from 2011 to 2015 was remarkable, with gains in both reading and math, in both fourth and eighth grades, and across all major racial groups. We can’t know for sure what accounted for this progress but it’s surely possible that the reforms of the Mitch Daniels/Tony Bennett era paid off. Did they continue on (then-governor) Mike Pence’s watch? And will any other states steal the spotlight—perhaps including Arizona?
5. Fiesta, forever, in Miami? Alberto Carvalho, the superintendent of Miami-Dade County, made a big splash this year when he accepted, and then rejected, an offer from Bill de Blasio to serve as New York City’s schools chancellor. His impressive progress on NAEP was a big part of his calling card; will it continue through 2017? He’s scheduled to speak at the release of the data next week, which could signal that the party will continue.
6. Is Chicago’s fifteen minutes of fame over? Chicago Public Schools have received numerous accolades of late, thanks to an analysis by Sean Reardon that examined “cohort gains” on state assessments and NAEP that showed tremendous movement as students progressed through elementary and middle school. But as I pointed out the other day, the district’s fourth graders have also made big gains of late, which ironically will make it harder for the district’s cohort gains to look as impressive going forward, at least on NAEP. I hope Chicago enjoyed its moment in the sun while it lasted.
7. Is the “California Way” working? Over the past decade, California has been a holdout from some of the big national reforms that most other states have embraced, especially clear and tough school accountability systems and test-based teacher evaluations. State board chair Mike Kirst has argued that the Golden State’s approach leads to more teacher buy-in and will prove itself in the end. And at least in a few subgroups, California has seen some progress in recent years. What will the 2017 data show?

For me, of course, there is one last question to ask: What will I do when I no longer have NAEP to obsess about? The 2019 results should be out soon enough—unless, of course, they go back to paper-and-pencil!

E.D. Hirsch, Jr. turned ninety years old two weeks ago. And the state of Louisiana has given the Cultural Literacy icon and architect of the Core Knowledge curriculum a belated birthday present. In a little-noticed press release issued Monday, the state’s Department of Education announced its plan, under a provision of the federal Every Student Succeeds Act, to develop and pilot a “streamlined English and social studies assessment….that align[s] with the knowledge and books taught in Louisiana classrooms.” Unlike most reading tests, which ask students to find the main idea or make inferences on reading passages about random topics, Louisiana plans to create test items on books and topics test-takers have actually studied in school and try them out in five school districts in the state, including two charter school networks.

If you’re familiar with Hirsch’s work, his many books including The Knowledge Deficit and The Making of Americans; if you understand the connection between background knowledge and language proficiency that he has championed for many decades, then Louisiana’s move is as obvious to you as it is overdue. For the uninitiated, University of Virginia cognitive scientist Dan Willingham in a recent blog post offered an economical summary of the Hirschean oeuvre:

Background knowledge is the main driver of language comprehension, whether written or spoken. Disadvantaged students are disproportionately dependent on schools to provide the background information that will make them effective readers because wealthy students have greater opportunity to gain this knowledge at home.

Simple. Intuitive. Obvious. At least until it’s time for students to sit for a standardized reading test. For years, Hirsch has made the case that such tests are fundamentally unfair to disadvantaged children, particularly low-income children of color, because of the nature of language itself. We may perceive reading comprehension as a content-neutral “skill” that can be taught, practiced, mastered, and tested in the abstract on any random topic, but this is deeply misleading. All reading tests are de facto tests of background knowledge, a point Hirsch made in a 2010 piece in The American Prospect, which I had the privilege to co-author, bluntly titled, “There’s No Such Thing as a Reading Test”:

Researchers have consistently demonstrated that in order to understand what you're reading, you need to know something about the subject matter. Students who are identified as "poor readers" comprehend with relative ease when asked to read passages on familiar subjects, outperforming even "good readers" who lack relevant background knowledge…Such findings should challenge our very idea of who is or is not good reader.

“The trouble is that by not requiring knowledge of any specific book or facts, reading tests have contributed to the false impression that reading is mainly about having skills such as being able to summarize, and not about background knowledge,” noted Lousiana’s state superintendent of education, John White. “By not requiring knowledge, tests create no incentive for particular knowledge to be taught.” This creates several kinds of mischief in schools trying their best to prepare students for high-stakes exams. When the knowledge demands of reading tests are unknown, it encourages teachers to devalue knowledge and prepare students by teaching comprehension “skills and strategies,” which are of limited value. And thus schools and teachers have no incentive to give disadvantaged kids what they need most: access to the broad general knowledge in science, social studies, the arts, and other subjects that advantaged kids are far more likely to encounter in their daily lives, and which are the true drivers of sophisticated language proficiency

In sum, what Louisiana proposes to experiment with is something of an assessment Holy Grail: a reading test that incentivizes the kind of teaching and learning disadvantaged students need to close pernicious achievement gaps in ELA. Keep that in mind next week when you hear the inevitably depressing results on NAEP reading.

The Louisiana proposal is a modest one, limited to five school districts and charter networks in the state. Modest, but portentous. For the first time, the education leadership of a U.S. state has demonstrated in its assessment policies a grasp of the foundational idea that English language proficiency is not a “skill” like throwing a ball or riding a bike that can be taught and tested in an abstract, content-agnostic way. The pilot signals the critical awareness that language-proficiency is knowledge dependent, and that educational equity is not served by ignoring this or trying to wish it away.

I suspect the pilot may get more attention for reducing the number of tests students take and for spreading them out over the school year, so that students are assessed immediately following a unit’s completion, leading to a cumulative score. But the longer-term win is to drive home the connection between broad general knowledge and broad general reading ability. Once established, that has the potential to have a dramatic impact by challenging the long dominant skills-and-strategies approach to reading instruction in favor of one that sees knowledge development in children—particularly disadvantaged children—as the indispensible Job One of reading instruction in American classrooms.

Bravo, Louisiana. And happy birthday, E.D Hirsch, Jr.

Massachusetts has earned well-deserved accolades for becoming America’s highest-achieving state, as measured by national academic assessments. But as Bay State leaders know, behind its accomplishments lurk some of America’s largest achievement gaps. And it’s not just that low-income and minority students do worse than their wealthier and white peers on average; there are also big gaps among high achievers, discrepancies that professor Jonathan Plucker rightly calls “excellence gaps.” One cause may be the state’s severe shortage of programs for gifted students.

The selection of Jeff Riley as Massachusetts’s new education commissioner is a perfect opportunity to do better by high-achieving students of color. His sterling record as a principal in Boston and superintendent in Lawrence suggests that he’s up to the challenge. But it won’t be easy.

According to the National Assessment of Education Progress, Massachusetts’s white, black, and Hispanic eighth graders reach the test’s highest level in math, deemed “advanced,” at twice the national rate. But this masks massive academic inequalities. Although one in every five white students is advanced, that’s true of just one in every twenty-five black and Hispanic students—a gap that is also twice what it is nationally.

Given this galling rift, you’d hope Massachusetts, so long accustomed to academic excellence, would try to lift up these talented students of color with programs that maximize their educations. But Fordham’s recent report, Is There a Gifted Gap?, which one of us coauthored, finds that it’s quite the opposite. A gifted program exists in just one out of every twenty Massachusetts elementary and middle schools, a lower rate than everywhere except Rhode Island and Vermont. And less than half of one percent of the state’s black and Hispanic students participate in one.

In place of these programs, Bay State schools claim to rely on “differentiated instruction”—a splendid notion wherein a single teacher is expected to tailor instruction to the individual abilities and prior achievements of each student in the classroom. The trouble is that it simply doesn't work in most classrooms because kids’ performance levels are too varied. Many teachers struggle heroically to cope, and some are reasonably effective. But most American teachers admit that they don't differentiate very well, certainly not on a sustained basis. Out of necessity, they triage most of their instruction to serve students who are struggling, leaving little time for those who have already met the standard and are capable of surging ahead.

This is particularly true for teachers in schools with lots of poor and minority learners, which further disadvantages the high achievers among them—the very Bay Staters who lag so far behind their more advantaged peers. And their schools face so many other challenges—attendance, discipline, nutrition—that attending to the educational needs of the brightest students is apt to fall by the wayside.

Far better would be to create effective gifted programs that are accessible to any student who meets their criteria, and then fill them by, first, using the results of the Massachusetts Comprehensive Assessment System to identify the top-scoring kids in every school. When Broward County, Florida, employed this approach, it worked exceptionally well for poor and minority children, report economists Laura Giuliano and David Card. Second, ask teachers to use a holistic approach to nominate additional kids who don’t earn top marks on the exam but who show uncommon potential.

This school-specific two-step model would give more students newfound opportunities to advance and help Massachusetts schools truly meet the needs of all students. It would also be inexpensive, yielding enough pupils to fill designated separate classrooms that provide enriched and accelerated curricula. Neither new staff nor new buildings would be required. And most importantly it would help erase the sorry excellence gaps that beset the Bay State, America’s academic poster child.

Massachusetts’s failure to do right by black and Hispanic high achievers robs the state and every community therein of tomorrow’s inventors, entrepreneurs, artists, and scientists. More vitally, it’s deeply unfair to the very children who depend most heavily on the public education system to do right by them. Leaders must do better. Here is hoping that Commissioner Riley just might.

On this week’s podcast, Hanna Skandera, former New Mexico education chief, joins Mike Petrilli and Alyssa Schwenk to discuss next week’s NAEP score release and why we’ve seen so little progress in recent years. On the Research Minute, Amber Northern examines how NCTQ’s ratings affect teacher education programs.

Amber’s Research Minute

Dan Goldhaber and Cory Koedel, “Public Accountability and Nudges: The Effect of an Information Intervention on the Responsiveness of Teacher Education Programs to External Ratings,” Calder (March 2018).

A new study by Susan Dynarski and colleagues examines the effects of a large for-profit charter school operator. Most existing charter school research is on non-profit operators with very little, if any, evidence at the K–12 level on for-profit outfits (although research findings on for-profit institutions at the college level are downright grim).

Analysts leverage randomized admissions lotteries from forty-four National Heritage Academy (NHA) charter schools in Michigan that were oversubscribed between 2002 and 2012 (NHA is the fourth largest for-profit operator in the country). The dataset includes roughly 27,000 applications, and analysts are able to observe at least one post-lottery achievement score for 90 percent of the students in the sample. Scores are observed through grade eight only, in part because most applicants are in kindergarten or first grade. Analysts are able to verify that the lotteries were indeed carried out randomly so they are essentially comparing the outcomes of lottery winners and losers.

Before we get to the findings, here’s a bit of background on NHA. Unlike other charters, almost half of NHA schools are located in suburban areas, forty percent in urban areas, and twelve percent in towns/rural areas. Their students are also less likely to qualify for free or reduced price lunch (61 percent versus 71 percent in other statewide charters) and more likely to be Asian, Hispanic or white than students enrolled in other charters in the state. The NHA schools are all brick and mortar—not online schools—focused on secular virtues modeled on Plato’s cardinal virtues (wisdom, respect, integrity, and so on), and use a “Dean Model,” which pairs an administrative dean—not the principal—with a team of no more than fifteen teachers to render instructional coaching and other supports.

The key finding is that attending a NHA charter school for one additional year is causally linked with greater test score gains in math (about 0.04 standard deviations), with smaller effects in reading that are not statistically significant. There are also no significant differences when it comes to attendance rates, classification as a special education student, and on-time grade progression. Yet when analyzing subgroup differences, they uncover a novel finding: Unlike other charter school studies that find the greatest benefits for low-income or minority students in urban areas, the benefits of attending NHA is concentrated among non-poor students attending their schools outside of urban areas.

Digging further, they attribute much of this difference to dosage in the treatment. In short, poorer kids who won the lottery accumulated just 0.4 more years in an oversubscribed NHA than those who lost the lottery. Specifically, the former are substantially less likely to enroll in NHA the very next fall compared to their advantaged peers. Why? Additional analyses show that distance may be a factor since poorer students are farther away from the nearest NHA school than non-poor kids (in fact, about a half mile farther on average), and as is the case elsewhere, charter schools are not obligated to offer transportation.

Finally, Dynarski and team survey faculty in NHA schools, other charter schools, and the traditional public schools that the NHA students most likely would have attended. Results show that NHA faculty devote more hours per week to math instruction, are more likely to group kids by their math ability, and generally provide teachers with more mentoring—all of which may help explain their key finding.

It’s not like we need another reason to say that charter schools can be a good option for all kids and that we need not restrain them to city limits. But this report provides one anyway. 

SOURCE: Susan Dynarski et al., “Estimating the Effects of a Large For-Profit Charter School Operator,” The National Bureau of Economic Research (March 2018).

The complaint that kids these days are addicted to their smartphones is a familiar refrain among parents and educators. While some schools try to pry iPhones from their students’ hands, others attempt to embrace the ubiquity of technology by integrating them into instruction with Bring Your Device (BYD) policies. Yet our understanding of how smartphones influence our ability to learn and focus is still limited. So let’s welcome a study by researchers at the University of Texas at Austin that examines how our smartphones affect cognition when we aren’t using—or even thinking about—them.

The analysts conducted experiments in which undergraduate participants completed a series of tests designed to measure their working memory, reasoning and problem solving, and sustained attention. The participating students were randomly selected to have their phone in one of three places: face down on the desk, in the testing room but in their pocket or bag, or outside the testing room. They also took a survey about the influence of their smart phones during the test and their personal smartphone dependence. 

The study finds significant differences in students’ capacities for working memory, reasoning, and problem solving based on the location of their phones. Participants whose phones were in the other room scored significantly better on the working memory test than those whose phones were in their pockets or on their desks. And students with their phones on the desk performed significantly worse than their peers on tests measuring reasoning and problem solving. Indeed, the only measure not affected by phone location was sustained attention.

In the survey, however, the majority of participants (75.9 percent) said the location of their phone didn’t influence their performance. This difference between perception and test results suggest that participants failed to anticipate the influence of their phones on their ability to perform cognitively demanding tasks. Additionally, the study finds that students with smartphone dependence one standard deviation above the mean saw much larger differences in working memory scores based on phone location. This suggests that those with a higher dependency on their smartphone are most affected by their phone’s presence—which means they also have the most to gain by putting their phones out of sight.

Although the study’s results are potentially worrying for educators, its conditions don’t necessarily reflect real classroom environments, in which other factors might mitigate phones’ influence. And the study was of undergraduates, so we can’t say for sure how it applies to middle and high schoolers. But it probably does, and the fact that simply having your phone nearby might limit students’ abilities to perform cognitively demanding tasks is cause for concern. It suggests that limiting in-school phone use—by, for example, requiring students to keep their phones in their lockers—could improve student learning.

SOURCE: Adrian F. Ward, Kristen Duke, Ayelet Gneezy, and Maarten W. Bos. “Brain Drain: The Mere Presence of One’s Own Smartphone Reduces Available Cognitive Capacity,” Journal of the Association for Consumer Research (2017).