The Education Gadfly Weekly

A small storm has blown up around the fact that certain math items on the 2015 National Assessment of Educational Progress (NAEP) do not align with what fourth and eighth graders are actually being taught in a few states—mainly places attempting to implement the Common Core State Standards within their schools’ curricula.

NAEP is only administered in grades four, eight, and twelve. So the specific issue is whether the fourth graders who sat for NAEP this spring had a reasonable opportunity to learn the skills, algorithms, techniques—broadly speaking, “the content”—on that test. If their state standards had moved some portion of what used to be fourth-grade math to the fifth or sixth grade, or replaced it with something else entirely, their state’s NAEP scores would likely be lower.

This kind of misalignment is blamed for some of the math declines that NAEP recently reported. Department officials in Maryland, for example, examined the NAEP math sub-scores and determined that many Maryland fourth graders are no longer being taught some of those things before they take the test.

We are left to wonder: Should NAEP frameworks and assessments be updated to reflect what’s in the Common Core, and perhaps other recent curricular developments?

Such questions arise periodically in all the subjects that NAEP assesses. They’re particularly acute in math, though, because the field is more “cumulative” than most disciplines; because the Common Core did make some significant changes in content emphasis and sequencing; and because math is one of only two subjects that are both included in the Common Core and part of the elaborate federal and state accountability structures that sit atop K–12 academic standards.

Although the NAEP math assessment is now given every two years, the framework on which it is based hasn’t changed much for grades four and eight since the early 90s. Achievement since then can therefore be reported as part of a continuous trend line stretching back a quarter-century. (The twelfth-grade math framework was changed in 2005—which, to use NAEP jargon, “broke the trend line” and started a new one, although some special analyses make it possible to connect the two.)

Just how closely aligned is the present NAEP math framework to the Common Core? A panel of experts convened by the American Institutes of Research (AIR) has just analyzed that very question and found “reasonable agreement.” That works out to 79 percent overlap in fourth grade and 87 percent in eighth, though the degree of alignment varies widely depending on the specific math topic. 

Changing a NAEP framework is a huge deal—expensive, contentious, and slow—as well as a major policy decision because of the issue of trend lines. Which brings us to the essential issue: Is NAEP intended to mirror what’s being taught today, or is it intended to monitor achievement over time? Is it an assessment of the efficacy of the present curriculum? Or should we use it as a barometer of how well today’s curriculum and instruction prepare students, as measured by a preexisting, unchanging standard?

Alongside “main NAEP,” there’s something called the “long-term trend NAEP,” the frameworks and content of which haven’t changed since the early 70s. That perspective is surely valuable to have. But because NAEP didn’t start providing state-specific data until the early 90s, we depend on main NAEP to know whether states and cities are doing any better—any better than before, any better in comparison with one another, and any better in relation to Singapore or Germany (once the statisticians work their magic and interconnect NAEP results with PISA or TIMSS).

This sort of sort of information is only knowable, obviously if the metric is stable. And that’s only possible when the tests don’t change. (The items change, of course, but not what’s being tested.)

As has no doubt become clear, I’m for stability here. It’s critical that NAEP’s math (and reading and writing) frameworks not flex with recent changes in standards, curriculum or pedagogical emphasis. That doesn’t mean they shouldn’t be reviewed from time to time, and it certainly doesn’t mean that NAEP is static; at the moment, for example, it’s in the middle of a complex and costly transition from paper-and-pencil to digital testing.

And stability is especially important today, because it’s never been more important that NAEP remain a trustworthy audit of American public education as it changes in all sorts of different ways. If the audit itself is revised to incorporate the Common Core, how will we know if Common Core states make greater gains than non-Common Core states? How will we know whether states and cities that faithfully implement their new standards are faring better than those that only pay lip service to them? How will we know whether places with lots of school choice are producing higher achievement than those without, or which of the state-specific accountability regimes likely to follow ESEA reauthorization are working best? How about correlations between spending and achievement? Between widespread reliance on technology and old-fashioned flesh-and-blood instruction? NAEP isn’t an experiment and therefore does not prove causation. But it’s the only reliable inventory of whether outcomes in American education are strengthening or weakening at the jurisdictional levels that matter most.

When we change the frameworks, we lose all that. It’s the same as replacing the yardstick, moving the goalposts, or sending in a different movie reviewer.

If every U.S. state and city were instituting the exact same procedures, then maybe we could constructively revise NAEP to align more precisely with the things they’re all doing. But when they’re reforming in so many different ways, NAEP mustn’t budge. The yardstick must remain thirty-six inches long.

(And that’s without even getting into the heinous politics that would surely follow if anyone made a serious move to remodel NAEP along the contours of the Common Core. Yikes—yet another effort by the federal government to ram the Common Core down everybody’s throats? There would be hell to pay before, during, and after the 2016 elections.)

So leave it alone. Listen to the president of the National Council of Teachers of Mathematics when she says to leave it alone. Accept the fact that the present NAEP math framework is still pretty much aligned with what cutting-edge math educators deemed most important twenty-odd years ago. Trust me, the math framework developed at that time was itself deeply controversial, the target of angry allegations that NAEP was getting “fuzzy” and rejecting traditional arithmetic. Understand that we broke the trend line for twelfth-grade a decade ago, which was controversial and probably unwise. Conduct as many studies as you like, but please do not stir this pot again now.

Creatas/Thinkstock

As a young child, Adrian was quick to anger and often acted out in class, sometimes physically. In fourth grade, his school classified him as having emotional problems and assigned him a personal aide. After a few years, the aide was phased out; his behavior improved, but the disciplinary consequences got worse. "If he lost his temper, he was generally suspended," recalls his mother, who asked not to be identified. "I had meetings upon meetings with the vice principals, but they would say, 'This is what we do; we have no money for things like detention or supervision for in-school suspension.'"

The barrage of disciplinary actions against Adrian (not his actual name) began to feel like harassment. "Countless suspensions for countless issues," his mother recalls. Before a six-month suspension, a lawyer told her that the school was "essentially a dictatorship" and that she had no real recourse. Frustrated and increasingly embittered, the family withdrew Adrian, moved away, and enrolled him in a public school where minor misbehaviors were punished with detention, not suspensions. "The school got rid of him by excessive penalties and suspensions," she concludes.

You might assume this is yet another tale out of Eva Moskowitz's network of Success Academy charter schools, which have been roiled in recent weeks by exposes in the New York Times and PBS NewsHour. Those stories cast a harsh light on the network's disciplinary practices, including allegations that one school had a "got-to-go" list of difficult students for the purpose of counseling them out. But Adrian was not a student at Success or another "no-excuses" charter school. He attended a regular district-run public school in Irvington, New York, a wealthy Westchester suburb where the median household income is nearly $100,000 and home prices routinely exceed $1 million.

Not far away, in the affluent suburban school district of Montclair, New Jersey, minutes from an August meeting show that the board of education approved spending nearly $5 million this year on a curious outlay. The money—an average of $63,000 per student—went toward tuition payments on "out-of-district placements" for seventy-nine children with a variety of classifications, including learning disabilities and "other health impairments." To be sure, there are often good reasons to place children out of district, and no district can serve all students equally well. But there aren’t always clear and obvious distinctions to be made between those kids who genuinely need alternative settings and those (like Adrian) who run afoul of the rules so frequently, or who are penalized so often and systematically, that they simply give up and leave.

Much is left to the discretion of school administrators, who face no small amount of pressure from parents to minimize disruption and maximize student achievement. In affluent communities, this would strike many as a matter of common sense. But it represents a form of privilege, it seems, in schools serving low-income, urban children.

"Success Academy works for some kids, but not for all kids," wrote education historian Diane Ravitch, a fierce and frequent Moskowitz critic. "Public schools are supposed to work for all kids. Granted, there are magnet schools and special schools, but there are supposed to be public schools where no one is turned away, no one is counseled out."

Echoing Ravitch, Hillary Clinton said over the weekend that some charters “don’t take the hardest-to-teach kids, or, if they do, they don’t keep them. And so the public schools are often in a no-win situation, because they do, thankfully, take everybody,”

Perhaps there are public schools that “take everybody.” But one thing is certain: If you are the bright son or daughter of affluent parents, chronic classroom disruption is foreign to your school experience. If you encounter it all, you can be confident that it won't last long. You almost never share a classroom with challenging, high-need kids. Wealthy families have any number of ways to insulate themselves from anything interfering with their children's education. There are public school administrators willing to marginalize and punish kids who act out, even for infractions beneath notice at chaotic inner-city schools. Affluent public schools hire tutors for those they suspend, or else pay tuition somewhere else for students they "lack the resources to adequately serve." And many districts do maintain the "magnet schools and special schools" Ravitch referenced, at least in part to accommodate such youngsters outside of "regular" public schools.

Then, of course, there is the most common tactic for sorting out the hardest to teach: the iron reality of the real estate market, which denies low-income families any hope of moving to affluent neighborhoods with "high-performing" public schools. "Regular" schools serving predominantly poor children, who face overwhelming challenges that affluent children never have to confront, are home to more than their share of behavior problems. It's what drives many urban parents to charter schools in the first place.

That brings us back to Success Academy and Eva Moskowitz. There may be no more divisive figure in American education. To her detractors (and they are legion), the New York Times exposé was the "smoking gun." To them, it "proved" that she achieves her eye-popping results by systematically shedding the hardest to teach—low-achieving children with behavior problems, disabilities, and difficulty understanding English. For her part, Moskowitz vehemently denies having a policy of pushing out difficult students and points to suspension rates in her schools that are lower than comparable New York City schools.

My object is not to condemn or praise Moskowitz. But let's suppose for a moment that the charges are true and that she achieves her extraordinary test scores by "counseling out" the kids least likely to produce such scores. As a thought exercise, consider this awkward bit of calculus: If counseling out some number of unruly, disruptive, or hard-to-teach students enables a school—any school—to bring twenty-five, fifty, or even one hundred times that number of other students to levels of achievement that "regular" schools have historically proven unable to equal—are you OK with that?

If your answer is no, then ask yourself if you're OK with it in places like Montclair and Irvington, Greenwich, or Bethesda. How do you feel about the fact that schools full of rich kids can and do make other arrangements for the hardest to teach without drawing the attention of the New York Times or PBS? Why does our definition of "fairness" and "equity" require that schools serving the less fortunate—and only those schools—must serve every child? If you are poor, black, or brown and bright in America, do you simply not deserve the classroom conditions taken for granted by those who sprang from wealthier wombs?

Are you OK with that?

Let's not kid ourselves that "creaming" and "counseling out" are rarities in American public education. But it's in rich neighborhoods, not poor ones, where such practices thrive. Let's not kid ourselves that those who pay a premium price for their children's education, whether in private school tuition or school taxes in well-off communities, don't demand and receive schools largely free of the hardest to teach.

One final question, perhaps the most uncomfortable of all: If all this creaming, counseling out, and ensuring just the right environment is a standard part of American education for so many, why does it become a problem—why does it make national news—only when someone gets caught doing it for poor black and brown kids?

Editor’s note: This article originally appeared in a slightly different form in U.S. News & World Report.

Creatas/Thinkstock

When is a test not a test? Sure, there’s an easy answer—“When it doesn’t send opt-out parents running for their torches and pitchforks”—but that’s not what we’re looking for. Give up? It’s when the test is a “locally driven performance assessment.” An article in Education Week explains the rise of these specially designed student tasks in eight New Hampshire school districts, which have been granted authority by the Education Department to employ them as alternatives to standardized tests. The districts will work with the state and one another to develop Performance Assessment of Competency Education (PACE), a series of individual and group queries that allow students to exhibit mastery over a subject without filling in bubbles. The challenges (which include the design of a forty-five-thousand-cubic-foot water tower to show proficiency in geometry) sound stimulating, and the Granite State’s record in competency-based education is extensive. It’s not hard to see why such an option would be attractive to state and local officials, especially when testing has become roughly as popular there as a leaf-peeping tax. What remains to be seen is whether this approach to assessment captures the same vital data as traditional measures.

Of course, some folks will never give up on the venerable paper assessment. For those happy few, who doubtless spend their evenings sharpening Number 2 pencils and reviewing the various and contradictory meanings of the word “sanction,” the Gadfly has a treat this week: The PARCC consortium has released hundreds of questions from its 2015 round of Common Core-aligned exams. The problems demonstrate both rigor and clarity, requiring that students explain their mathematical rationales and compare multiple texts. The release will hopefully demystify the new tests for parents, many of whom have felt alienated by unfamiliar question construction. Now they can determine once and for all whether they’re smarter than a fifth grader.

Finally, California administrators are evidently facing a far more difficult test than anything PARCC or New Hampshire educators could have dreamt up. It’s the assignment of restoring the Los Angeles Unified School District to fiscal sustainability, and if last week’s Los Angeles Times scoop is any indication, the challenge has thus far proven insurmountable. According to a panel of budget experts assembled by Superintendent Ramon C. Cortines, the district is presently facing a $333 million deficit for the 2017–18 school year, which will nearly double in the next two years. As is often the case, the shortfall is driven predominantly by swiftly accumulating pension costs. But the crisis is further aggravated by sinking enrollment in public schools as families embrace the city’s charter options. Local officials, as well as anyone with an interest in keeping our school system out of the red, would be well advised to consult Fordham’s own report on insolvent school districts for more on the subject.

Atlanta cheating, Eva’s Success Academies, poverty and brain science, and measuring Common Core’s effects. 

Amber's Research Minute

In spite of some well-publicized controversies, performance-based teacher evaluations have maintained a strong presence in most states. A new report from the National Council on Teacher Quality (NCTQ) examines the policy landscape of teacher and principal evaluations, as well as various states’ successes in using evaluations to inform teacher practice and administrative decisions.

As of 2015, twenty-seven states require annual evaluations for all teachers, and forty-five require annual evaluations for all new, probationary teachers. Forty-three states require objective measures of student achievement to be included in teacher evaluations; seventeen use student growth as the “preponderant” criterion for evaluations; and an additional eighteen count growth measures as “significant” criteria.

Despite these new policies, however, a “troubling pattern” lingers on from the evaluation systems of yesteryear: The overwhelming majority of teachers are still labeled as “effective” or “highly effective.” NCTQ notes this could be the result of several factors, including the fact that few states utilize multiple observations and multiple observers—which is problematic because many principals are either unable or unwilling to “make distinctions about teacher skills” when conducting observations. In addition, Student Learning Objectives (SLOs)—which are required or allowed by twenty-two states—fail to effectively differentiate teacher performance. According to NCTQ, these objectives are often a poor measure of teacher performance because teachers either lack the data and assessment training to select the best achievement goals or are indirectly incentivized to set low goals. (The other problem is that managers won’t give bad reviews to staffers they can’t fire, which is still the case in the vast majority of schools.)

Thirty-four states also require annual evaluations for all principals. Despite this majority, many principal evaluation policies lack important features. For instance, policies in twenty-two states do not identify who is responsible for conducting principal evaluations, and only twenty-seven states actually require principal evaluators to receive training. Most worrisome, there are ten states where teachers—but not principals—can be dismissed based on evaluation results. Given the monumental importance of effective school leadership, it’s inexcusable that only teachers would be held accountable for their evaluation ratings.

NCTQ labels Delaware, Florida, and Louisiana as “leading the nation” in using evaluations to shape teacher training, professional development, improvement planning, compensation, and accountability.

NCTQ rounds out its report with some solid recommendations, which include aligning teacher and principal evaluations and focusing on incentivizing states that are willing to implement evaluation policies rather than “twisting the arms” of unwilling states. The federal government ought to keep these suggestions in mind when it considers doling out the next round of federal waivers. 

SOURCE:  Kathryn M. Doherty and Sandi Jacobs, “State of the States 2015: Evaluating Teaching, Leading, and Learning,” The National Council on Teacher Quality (November 2015).

Success for All specializes in whole-school turnaround for struggling elementary schools. Its 2010 proposal for an Investing in Innovation (i3) grant called for the program, whose primary goal is to ensure that every child learns to read well in elementary school, to grow from one thousand schools to more than two thousand. The Baltimore-based organization was one of only four to grab the shiniest brass ring in the i3 competition—a five-year, $50 million “scale-up” grant. Teach For America, KIPP, and the Reading Recovery program snared the other three.

This third and final report from MDRC looks at SFA’s impacts between kindergarten and second grade in five school districts over a three-year period covered under the i3 grant. A total of thirty-seven schools across five school districts were part of the study—nineteen randomly chosen to implement SFA, along with eighteen control schools that either stuck with their existing reading programs or choose new ones other than SFA.

The report finds that SFA is “an effective vehicle for teaching phonics,” showing statistically significant effects for second graders who were in SFA for all three years. SFA students also performed better than the average in reading fluency and comprehension, though not significantly. The report authors don’t say so, but the relative lack of traction on comprehension would seem to be of minor concern for K–2 since comprehension tends to be cumulative in nature—as vocabulary and background knowledge grows, so does comprehension. The most encouraging finding is SFA’s effectiveness with students who enter school with low pre-literacy skills. “Second graders in the average SFA school who had started kindergarten in the bottom half of the sample in terms of their knowledge of the alphabet and their ability to sound out words registered significantly higher scores on measures of phonics skills, word recognition, and reading fluency than similar students in control group schools,” the authors note. This is an especially noteworthy finding because, as MDRC notes, students who begin school already behind are of special concern to policymakers and practitioners. Neither is SFA a budget buster: The report calculates direct expenditures for adopting SFA’s model at $119 more per student, per year in SFA schools than in control group schools. Not a bad bargain for either schools or the taxpayers who footed the bill for SFA’s i3 grant.

SOURCE: Janet Quint et al., “Scaling up the success for all model of school reform,” MDRC (September 2015).

Earlier this year, the RAND Corporation surveyed the 3,338 principals with current Teach For America (TFA) corps members at their schools. These principals are, on average, slightly less experienced and more racially diverse than American principals at large—and far more likely to run a charter school (27 percent work at charters).

In general, the survey’s results suggest that most principals who work with TFA corps members view them positively. Eighty percent of respondents said they were satisfied with the corps members at their schools; 86 percent said they would be willing to hire another corps member; and 66 percent would “definitely recommend” doing so. Moreover, a majority of respondents said corps members were at least as proficient as other novice teachers across a range of skills, including developing positive relationships with colleagues and administrators, having high expectations for students, and improving student performance. And 87 percent of respondents said they were satisfied with the support TFA provides, which three-quarters agreed complemented their school’s induction or training.

Despite these generally positive findings, the survey identified two areas of concern: First, half of the respondents identified weak classroom management as a reason not to hire additional TFA corps members. Second, 57 percent said that TFA’s short two-year commitment was a disincentive to hire.

Interestingly, both TFA alumni and principals of charter schools viewed corps members and the program more negatively than other principals. For example, both groups were less satisfied with the supports provided by the program, and both also reported that corps members had less subject matter expertise than other novice teachers. Nevertheless, charter school principals were more likely to say they would hire additional corps members than other principals—perhaps because they face tighter budgets and fewer hiring restrictions.

Some of the differences in perception captured by the survey may reflect respondents’ expectations rather than teachers’ skills. For example, charter leaders and TFA alumni may expect more of corps members than other principals, leading them to view their performance more negatively. Then again, there could be real differences between teachers hired by these groups and other teachers. Perhaps, for one reason or another, the best corps members are unlikely to be assigned to charter schools. Or perhaps novice teachers in charters are more knowledgeable than their conventional public school peers, making corps members assigned to charters comparatively less so.

Regardless of the true explanation, two things seem clear from the survey. First, despite all the fuss, principals don’t hate TFA. And second, they’re nobody’s fools. Pretty much everyone knows that many of these talented young folks will flee the classroom when the overwhelming challenges of the job become apparent. Unfortunately, what nobody has really figured out (still) is how to convince them to stay.

SOURCE: Mollie Rudnick, Amanda F Edelman, Ujwal Kharel, and Matthew W. Lewis, "Results from the Teach For America 2015 National Principal Survey," RAND Corporation (October 2015).