High Expectations
Heartbreak on NAEP
As disappointing as these results are, let’s not jump to conclusions. Michael J. Petrilli
Heartbreak on NAEP
Unfortunately, the rumors, predictions, and surmises were correct: Scores on the National Assessment of Educational Progress (NAEP) are mostly down or flat. The worst news came in eighth-grade math, where twenty-two states saw declines. One of the only bright spots is fourth-grade reading, where ten states (as well as Washington, D.C., Boston, Chicago, and Cleveland) posted gains.
Why this happened will be combed over and argued. So far, it feels like anyone’s guess (more on that below). But there’s no denying that it’s bad news. It had come to seem like NAEP scores would always go up, at least over the long term, just like it had come to seem like murder rates would always go down. Now the real world has intervened to remind us that social progress is not inevitable. Let’s not sugarcoat it: This is deeply disheartening for our country, our K–12 system, and especially our kids.
As our friends in the research community like to remind us, it’s impossible to draw causal connections from changes in NAEP data; doing so is “misNAEPery.” Yet we can’t help but search for explanations. And we can certainly float hypotheses about the trends—educated guesses that can then be tested using more rigorous methods (like these).
What might be going on? It could certainly be something happening inside our schools. Maybe the transition to the Common Core is causing disruption and growing pains (or worse), and those are reflected in these data. Maybe the political debate over standards, testing, and teacher evaluations has caused uncertainty in the classroom or discouraged kids from trying as hard. Maybe Arne Duncan’s waivers relieved the pressure on schools to boost achievement, and they consequently took their foot off the gas. Some states will explain that they altered the portions of English language learners and students with special needs who were excused from NAEP testing. All plausible.
But it’s also plausible that these trends reflect something going on outside of schools—namely, the economic condition of our country and our communities. As I argued the other day, the Great Recession and its aftermath could have acted as a stiff headwind. As schools face more challenging demographics—partly because of the decades-long surge in immigration, but also because of the economic dislocation facing many students and their families—they have to work harder just to stand still.
Do the data bear this out? Looking at state-level differences, my read is…not really. Perhaps Nevada, Rhode Island, Florida, North Carolina, Ohio, and Kentucky can blame the recession. But it’s fair to ask why the bad economy didn’t knock Georgia, Michigan, Indiana, Alabama, or Arizona off-course. And North Dakota and Texas had better find another excuse for their test score drops.
Trends in median family income versus eighth-grade mathematics scores
(Change in family median income in parentheses)
* Data are from Kids Count; I then used the Bureau of Labor Statistics’s inflation adjuster calculator to determine inflation-adjusted income. I considered the area between -2% and +2% to be flat.
Some are already using these results to attack education reforms, including Common Core, charter schools, vouchers, teacher evaluations, and anything else they don’t like. And in fairness, some reformers would have committed misNAEPery to bolster their arguments had the trends gone in the other direction. The most honest approach today is to urge patience—to reserve judgment until more sophisticated analyses emerge and wait for 2017 to see if these numbers are a one-time blip or the beginning of a disturbing trend. We certainly don’t have enough information now to justify a major change of course. But that day may yet come.
pixelheadphoto/iStock/Thinkstock
President Obama's meaningless pledge to reduce school tests
Over the weekend, President Barack Obama received high praise from parents and teachers for acknowledging that testing is taking too much time away from teaching, learning and fostering creativity in schools, and recommending that standardized tests take up no more than 2 percent of total school instructional time. Frankly, this is arrant nonsense.
From time to time, I'm asked to give a talk about education. If I look at how I spend my time over the course of a year, giving presentations and speeches is a very small part of my job—less than 2 percent. However, if my effectiveness were to be judged on the audience response to the handful of talks I give each year, I'd spend a lot more time writing and practicing speeches. I'd fret endlessly over my PowerPoint slides and leave-behinds. I'd sprinkle in more jokes to be entertaining; I'd probably say whatever I thought would get audiences to like me more, rather than challenging my listeners. I'd definitely spend a lot more on suits and dry cleaning than I do now.
But most critically, I'd spend far less time on all the other things I do—writing, reading, teaching and learning, visiting schools to stay current—that might make any talk I give worth listening to in the first place. In short, if this minor part of my job, however useful, were to become the alpha and omega of how my effectiveness was measured, it would quickly change nearly everything else about my work. And not necessarily for the better.
It's the same with testing. First of all, reports that Obama "plans to limit standardized testing to no more than 2 percent of class time," are simply wrong. The federal government has virtually no say about how much time schools spend testing. The vast majority of tests that our children take are driven by states and school districts, as well individual schools and teachers—not by Washington. The best the president can do is use the bully pulpit to encourage less testing, and even then, there's reason to be skeptical.
The amount of time kids spend on testing is not the issue. It's what the tests are used for that matters. Like my speech example, when you use standardized tests to make high-stakes judgments about schools and teachers, they are no longer a mere diagnostic. The testing tail wags the schooling dog.
"I still have no question that we need to check at least once a year to make sure our kids are on track or identify areas where they need support," said Education Secretary Arne Duncan, thereby demonstrating that the administration doesn't understand or thinks it can elide the effect of testing on schools. If there are stakes attached to those tests for schools or teachers, then there will be little or no change in current classroom practice.
To be clear, I am not anti-testing. Far from it: The data from tests are some of the most valuable intelligence we can access in the struggle to improve our education system. Test data have created the demand for school choice and charter schools. They supply the proof point demonstrating how poorly we've historically served low-income children of color. The field of education simply cannot advance without testing; it provides one of the few empirical bits of data we have to give insight into the effectiveness of changes in any number of school-based inputs, from curriculum to education technology.
But the moment you set out to trigger corrective actions and interventions using tests (which are, after all, designed merely to measure student performance), you are fundamentally shifting their function from providing evidence of student performance to something closer to the very purpose of schooling. This is precisely what has been occurring in our schools over the last decade or more. When parents complain about over-testing, what they are responding to is not the tests themselves—which take up a vanishingly small amount of class time—but the effects of test-and-prep culture, which has fundamentally altered the experience of schooling for our children.
The Obama talk on testing seeks to curry favor with parents and teachers (and their unions) while doing nearly nothing to change the fundamental role of testing and its effect on schooling. It's all well and good to "encourage" states, districts, and schools to limit testing. But as long as test-driven accountability measures are used trigger corrective measures in schools instead of providing feedback to parents and other stakeholders, it won't matter if children take two tests or two thousand; the effects will be the same. And here, Obama and Duncan are on the horns of a dilemma: Testing is retained in both the House and Senate versions of the Elementary and Secondary Education Act currently up for reauthorization. But if you do away with it, there will be hell to pay among key Democrat constituents worried (rightfully) about equal educational opportunity for minority children and families.
It is nearly irrational to be "anti-testing." Education cannot be a serious discipline without the sunshine created by testing data. But our present relationship with testing is like holding a wolf by its ears: We don't like it, but we can't let go. The president's—and Duncan's—remarks this weekend get us no closer to rectifying our relationship with testing.
Editor’s note: A slightly different version of this article originally ran in U.S. News & World Report.
Scott Olson/Getty Images News/Thinkstock
The administration wants it both ways on testing
President Obama and Secretary of Education Arne Duncan deserve credit for acknowledging this weekend that there’s too much testing in our schools today and that “the administration bears some of the responsibility.”
Indeed it does. That’s because its decision to condition ESEA flexibility on state adoption of teacher evaluation systems has not only raised the stakes of reading and math tests (making them less popular and potentially more damaging to the educational enterprise). It’s also led to a proliferation of tests in “non-tested subjects”—everything from P.E. to social studies and beyond—for the sole purpose of collecting data to judge teachers’ effectiveness.
Yet, as Matt Barnum argues persuasively at the Seventy Four, the feds aren't willing to actually fix this problem:
The new report did not capture a precise measure on what proportion of tests were required by teacher evaluation, but it does point out that many states have put in place new assessments “to satisfy state regulations and laws for teacher and principal evaluation driven by and approved by U.S. Department of Education policies.”
But an initial reading of the department’s guidance suggests it is sticking to these policies: “The Department will work with states that wish to amend (their federal) flexibility waiver … while still maintaining teacher and leader evaluation and support systems that include growth in student learning.” (emphasis added)
The problem the administration faces is that it is incredibly difficult to measure a teacher’s contribution to student growth in the absence of a standardized test.
The administration is committing a cardinal sin of policy making by demonstrating an unwillingness to acknowledge trade-offs. This is one of those instances in which you can’t have it all: Either you can reduce testing, or you can continue to demand test-based teacher evaluations in all subjects. It’s one or the other.
Contrast this approach with Ohio’s. Last year, State Superintendent Dick Ross published a report on the testing load in the state’s schools that showed strikingly similar results as the new Council for Great City Schools study. It found that about one-quarter of the testing in the Buckeye State was linked solely to the need for data for teacher evaluations in subjects other than math and reading. To his credit, Ross proposed that districts simply dump those tests. He made a choice, in other words.
Regrettably, the Ohio General Assembly did not go along with his recommendation—but for an understandable reason. Because of Ohio’s federal waiver, Buckeye State districts couldn’t just move to evaluations based on teacher observations and the like. If they had gotten rid of excess tests, they would have had to use reading and math scores to evaluate all teachers—gym teachers, art teachers, the whole crew. This is quite obviously inane, and it demands a change in federal policy.
The Obama administration is trying to have it both ways. It wants fewer tests but isn’t willing to give up on test-based teacher evaluations. Meaning that, alas, it has failed this test.
eabff/iStock/Thinkstock
The lost boys
- The plight of boys in school, and particularly boys from underprivileged backgrounds, is a story we’ve all gotten used to hearing about. Whether in our home state of Ohio or in international educational utopias like Finland, young men are lagging behind their female classmates in testing performance, high school graduation, college attendance, and a slew of disciplinary issues. The New York Times’s Upshot blog has an article on the academic gender gap that delves into the existing research to reach a distressing conclusion: The cumulative stress of all social hindrances, from domestic violence to parental absence to poverty, impose a greater toll on boys than girls. Although girls in every setting are more likely than boys to be adequately prepared for kindergarten (and less likely to rack up absences once they begin), their comparative edge grows much wider within deprived populations like immigrant and low-income families. With forces like this arrayed against them, perhaps it’s no wonder that these boys are underrepresented in selective high schools and colleges.
- Let’s take a moment to lavish some praise on a state whose merits are too seldom celebrated: Delaware. Last week, the First State became just the sixth state (wait, what?) to release detailed report cards on the quality of its teacher preparation programs. The assessments, which grew out of a comprehensive 2013 overhaul of Delaware’s teacher preparation sector, rate education schools on a 1–4 scale based on diversity and selectivity, how long program graduates spend teaching in the state, and how well those teachers’ students perform academically. The mighty Wilmington University and the University of Delaware earned top scores this year, although many smaller schools produced too few graduates to be fully assessed. The programs themselves aren’t too happy to be publicly grouped in tiers (imagine that), but teacher quality is way too important to go without vital data like these.
The NAEP is falling edition
Petrilli and Pondiscio discuss the fallen NAEP scores, debate the meaning of Obama’s pledge to reduce testing, and ponder school dress codes. Amber takes a look at NAEP’s alignment with Common Core math.
Amber's Research Minute
SOURCE: Phil Daro, Gerunda B. Hughes, and Fran Stancavage, "Study of the Alignment of the 2015 NAEP Mathematics Items at Grades 4 and 8 to the Common Core State Standards (CCSS) for Mathematics," AIR (October 2015).
Transcript
Mike Petrilli: Hello. This is your host Mike Petrilli, of the Thomas B. Fordham Institute. Here at the Education Gadfly Show, and online at edexcellence.net. Now please join me welcoming my co-host the DJ Silento of Education reform. Robert Pondiscio.
RobertPondiscio: Watch me whip, watch me NAEP NAEP?
Mike Petrilli: Watch me NAEP NAEP. That's right! We've seen it on Twitter. Watch me whip, watch me NAEP NAEP. Hey, my second grader loves this song, especially the Superman part. Watch me Superman. Yes, NAEP NAEP, how perfect. This is our special all NAEP addition of the Education Gadfly show Robert. It's NAEP day, should be a national holiday.
RobertPondiscio: It is a national holiday, if you work at the Fordham Institute.
Mike Petrilli: Yes, that's right.
RobertPondiscio: One or two other places, but we're wonky like that.
Mike Petrilli: We are wonky. Used to be a time there were only a few of us organizations out there analyzing the NAEP results. Some would argue committing mis-NAEP-ry, but now there's all kinds of groups involved, and more scrutiny than ever, so we have to stick to the facts Robert. This is not causal data here.
RobertPondiscio: Watch me wonk, watch me NAEP NAEP.
Mike Petrilli: Good. See impressive. This from a man who has been staying up late watching his Mets get beat.
RobertPondiscio: Oh you had to work that in there, didn't you Mike? Go ahead.
Mike Petrilli: Let's play Pardon the Gaff. Clara get us started.
Clara: What do we make of the new NAEP scores? Are the declines bad news for Ed reform?
Mike Petrilli: Robert? You going to try to sugar coat this one Robert?
RobertPondiscio: When have I ever sugar coated anything Michael? Can we get out of the business, please, of ... how do I want to put this ... Of changing our minds and over reacting to ever little twitching nerve impulse that comes out of the data. Come on, let's look at some long term trends. Here's the best thing I read this week, and I read this before NAEP came out.
Mike Petrilli: I did.
RobertPondiscio: A nice report. I can't remember the title of it. From the Urban Institute looking at 10 year trend datas. One of the biggest complaints of mis-NAEP-ry which you talked about. Is that why don't we look at the demographically corrected versions of NAEP, we never do. Well matching [inaudible 00:02:22] has made it easy for us. We don't ever have to have this conversation again, and to me everybody was talking about how Texas and Florida look a lot better. They're right up there with Massachusetts when you look at through this prism, but the thing that nobody's talking about is all 50 states, if you look at 10 year trends are improving. That to me is the big news, and that happened even before this came out.
Mike Petrilli: Well that did happen before. One of the things that makes these scores so disheartening is that in math especially, we have seen for almost 20 years now, nothing but up, up, up. It's kind of like the crime rates. We're used to them going down, down, down, and when we. Hold on. When we see a little blip, it makes us nervous. I totally agree.
RobertPondiscio: Move away from the window, Mike. It's going to be okay.
Mike Petrilli: We can not make any conclusions. We got to see what happens next time.
RobertPondiscio: Right.
Mike Petrilli: We certainly need more analysis, but look you'd rather see tests scores up then go down. You'd rather see crime stats go down then go up, and when you see these trends, perhaps take a wrong turn, it's something that makes you nervous. All I'm saying is, of course we can't draw conclusions, but let's also not sugar coat it. Hey let's be honest. Some of our friends, 2 years ago, were out there crowing about the fact that the District of Columbia and Tennessee saw big gains.
RobertPondiscio: Yup.
Mike Petrilli: Claiming it was because of their implementation of Common Core and Teacher Evaluations.
RobertPondiscio: I just read and interesting op-ed, and I can't remember where I read it. It was written by a guy, it was an Italian last name, like Petrillii? Something like that.
Mike Petrilli: Yup, yup, yup, yup. It's a good one.
RobertPondiscio: It was good. It was about how.
Mike Petrilli: That's my cousins.
RobertPondiscio: Was it.
Mike Petrilli: Yeah.
RobertPondiscio: Okay. It was really well written. It was written by another guy who wrote, another Italian name I think. Anyway, it was about how Common Core test results were showing. Don't shoot the messenger.
Mike Petrilli: Yup.
RobertPondiscio: They're going down because all these changes going on in classrooms. Why did we think that when we're asking teachers to do so much more, change their practices in the last couple of years. Why did we think we wouldn't see a down turn?
Mike Petrilli: Right.
You're parodying Secretary Duncan's line. He says that.
RobertPondiscio: I'm parodying nothing. I'm my own man.
Mike Petrilli: You're just an army Duncan lackey. That's fine, but look. There's an implementation dip. A well known phenomenon. That may be true. Nobody knows, but hey, we would rather see these scores going up. Let's hope that next time that is the case, and let the true wonk's dig in, do some more sophisticated analysis.
RobertPondiscio: If a frog had wings, Mike, it wouldn't bump its butt on the ground. Come on.
Mike Petrilli: Okay. I have no idea what that means Clara. Why don't we go onto Topic Number 2?
Clara: What's our take on President Obama's plan to reduce testing in schools.
RobertPondiscio: This is the one where you just try to irritate me, isn't it?
Mike Petrilli: Well you are just irritated, Robert, this week. Now on this one you're not an administration lackey.
RobertPondiscio: Oh my god.
Mike Petrilli: You actually seem to disagree with them on this.
RobertPondiscio: Vehemently. I was actually kind of ... I would say Obama and Secretary Duncan sort of ruined my weekend on this one. This is just nonsense. I wrote a piece in U.S. News about this, this week. This idea that you're going to reduce the number of tests. Let's count the number of ways in which this is silly.
1 - How many tests are federally mandated Mike?
Mike Petrilli: Well depends on how you count them. 3 to 8. Math and reading, plus science, it comes out to something like 24, I think.
RobertPondiscio: In any given year, 2? Let's be not disingenuous. The over test ... look I think is a real phenomenon, and we have to be mindful of that. That is not federally driven. By the way, did my brain fall out of my head?, or do Obama and Duncan have virtually nothing to say about the number of tests that schools, and states, and districts assign. Right?
Why are they even talking about this? The real mischief here is the effect of testing. I've been a classroom teacher. I'm still a classroom teacher. Let me tell you if you're not paying attention to the effects of testing, not the number of tests, but the fact that the test and prep culture we have materially changes the experience of schooling for our children. That's the problem. It's not the number of freaking tests.
Mike Petrilli: So Robert. I hate to break it to you, but you're wrong and you're wrong, twice.
RobertPondiscio: Oh man.
Mike Petrilli: All right. So here's the first thing. This is being driven in part by the federal government, not just the requirements of reading and math tests, but especially the new teacher evaluation mandate.
RobertPondiscio: That's what I'm saying Mike!
Mike Petrilli: Well no. You're talking about the stakes and I agree with that part.
RobertPondiscio: That is the stake.
Mike Petrilli: I'm also saying that quite literally there is now more testing happening because there's a mandate to collect student achievement data. Including in the non tested subjects. What have states and districts done? They've gone and created pre and post test for gym class!
RobertPondiscio: Okay, but Mike come one.
Mike Petrilli: It's ridiculous.
RobertPondiscio: Whoa. If you take away every single test, except the annual test in math and English, you will have no material change in the way schools are run right now.
Mike Petrilli: No, but Robert, the point is this. Why are we having this big backlash right now? I think, part of it is, a bunch of new tests have popped up in just the last couple of years. Including in gym class, and it's because of the teacher evaluation thing. Which is ridiculous.
RobertPondiscio: Right. Do you think if that goes away that nobody's going to complain about tests anymore?
Mike Petrilli: No. Listen. All right. It should go away. Now the problem was, states like Ohio would like to get rid of those things, but then they're told by the federal government. Well then what you have to do for gym teachers, rather than just, I don't know, go watch them and see if they're good teachers. If the kids are doing push ups correctly, or whatever they do. No, no, no. Instead you should start to evaluate them based on reading and math scores.
RobertPondiscio: That's just silly.
Mike Petrilli: Right. That's ridiculous. This is because the federal government with ... By the way a condition on waivers that is illegal, and that by the way nobody in Congress supports.
RobertPondiscio: It's really, really simple.
Mike Petrilli: So don't let your friends over there in the administration, Mr. Lackey, off the hook so easily. Now the second thing is, on your whole stakes thing. Look, Robert, I share a lot of your concerns about the design of accountability systems, or the way we are doing this has not encouraged good practice. We got to fix that.
RobertPondiscio: Correct. Correct. Stop right there, you were done.
Mike Petrilli: However, that doesn't mean throwing the baby out with the bath water, or else you become Diana Ravitch.
RobertPondiscio: Okay. No, no, no. Look.
Mike Petrilli: What is the difference between your position and Diane Ravitch's position? I just want to know!
RobertPondiscio: I'm all for accountability. Here's what I say. I don't have to have the solution to identify the problem. The problem is our relationship with testing is just out of wack. We are asking these tests to do things they were not intended to do. I want the data, because the data gives us the engine that creates the appetite for choice, for charters, for all these things that you and I love. If we are blind to the fact that we are making a toxic environment around these tests to serve our needs, we are going to lose this battle. We will have neither test, nor accountability.
Mike Petrilli: Okay, thank you Dr. Ravitch. Okay. Topic Number 3.
RobertPondiscio: Why do I subject myself to this?
Mike Petrilli: I don't know, because our listeners enjoy it. Okay, Topic Number 3.
Clara: Some students feel like school dress codes unfairly crack down on girls for wearing, quote on quote, "distracting clothing." Are dress codes sexist?
RobertPondiscio: Now you're just messing with me!
Mike Petrilli: I should say Robert, I did lie. I said it was going to be all NAEP, but I couldn't help but pass this one. This has gotten some attention that these dress codes. Look here's the argument, Robert, is that some of these dress codes, rather than be gender neutral, they say things like. Girls may not show their shoulders, let's say, if they have some kind of strapless dress on, or whatever, and there's nothing in the dress code about boys showing off their shoulders. It should at least be gender neutral. If you're going to wear a skirt, whether you're a girl or a boy, here's how long it can be. People thinking that some of these dress codes are out of date. Is this something the American public should be concerned about?
RobertPondiscio: Actually for what it's worth, this is one of those things because everybody has a feeling about this. Parents this is an intuitive issue, everybody has strong feelings about this, right? This is something that you're going to have to be an expert to have a strong opinion about. Look I like school uniforms for exactly this reason, it solves this problem. It takes the bat out of the hands of parents and kids, and say that I work in a charter school, we have uniforms, great, problem solved.
Mike Petrilli: Right.
RobertPondiscio: That's my answer. I mean on the one hand this issue.
Mike Petrilli: So you really don't believe in parent power do you? That's very interesting. You just took the bat out of the hands of the parent. I'm just kidding. In this case, right. Instead of having a dress code that says well you can buy something you bought at the mall, but it's got to meet these specifications. Instead you say you have to have a uniform.
RobertPondiscio: The funny thing is that in most schools, many schools, that have uniforms. What I would argue, they do it precisely backwards. If you're in elementary school you wear a school uniform, in middle school you wear half uniform, in high school you have a dress code. They should do it just the opposite. Let little kids wear whatever they want, because they're not going to wear distracting, and I'm making air quotes everybody, distracting clothes.
Mike Petrilli: Right. Well now this is the other thing about distracting. The other argument is that these dress codes are all about the boys. That they are basically [sent 00:11:04] on this thing that teenage boys are going to be sent into these crazy fits if the girls, young women maybe we should say, are scantily clad. Therefore we must make the girls cover up. We might as well have them wear one of those, what do they call them the hijab's, like it's over, right, the burka's, right.
RobertPondiscio: Right.
Mike Petrilli: Like we're over in the middle east, lest bad things happen. Again, if that is the case we should also make the boys cover up, because look what could happen the opposite way.
RobertPondiscio: Here's what I'm going to say. There are so many problems that we have in schools right now, this is one I have a difficult time getting exercised about. I say that as a father of a volleyball player that plays her sport in spandex, and that upsets me, and beach volleyball players had to wear bikinis.
Mike Petrilli: What do you want her wearing?
RobertPondiscio: Anything, a uniform.
Mike Petrilli: Well that.
RobertPondiscio: I mean I'm sympathetic to this issue, but.
Mike Petrilli: A Scottish kilt.
RobertPondiscio: The biggest issue that we face in American education.
Mike Petrilli: I know, but look here's the thing. I think we still should give schools a lot of leeway on this one.
RobertPondiscio: Sure.
Mike Petrilli: I think that.
RobertPondiscio: They should be able to choose the color of the uniform.
Mike Petrilli: We have pushed back against the adult authority in the schools on too many occasions. It all started with that crazy Tinker decision in the Supreme Court that said that kids have rights. Kids don't have rights!
RobertPondiscio: Not in my home.
Mike Petrilli: Okay. When you're 18 you get rights. Until then, forget about it. I'm glad we finally agree on something.
RobertPondiscio: We had to eventually.
Mike Petrilli: Yes. All right.
RobertPondiscio: Now can we agree that this is done?
Mike Petrilli: It is.
RobertPondiscio: Can I leave now, Mike?
Mike Petrilli: Done. You may leave now, but not you listeners because this everyone's favorite segment. Amber's Research Minute.
Amber welcome back to the show.
Amber: Thank you, Mike.
Mike Petrilli: We're ready to watch you Wonk, watch you NAEP NAEP. You really like that don't you. Maybe we do need to cut a video about that. What do you think?
Amber: Well imagine that. I have a new study out by the NAEP validity studies panel this week.
Mike Petrilli: Yes!
Amber: It analyzes the alignment of the 2015 math items. This is great for NAEP to the Common Core standards formats. The study classifies the actual 150 items at each grade level that are in the 2015 NAEP math item poll. These are the real items, 150 of them on NAEP, okay?
Panelists are to tag each NAEP item to a Common Core standard, or determine that it does not match a Common Core standard. Okay, it's simple enough. 18, I had to look a little bit I wanted to see who they got, but 18 mathematicians, teachers, math educators, supervisor with familiarity of the Common Core. Okay, so 18 folks looked at this stuff and their different backgrounds. The bottom line, reviewers determined that the concordat’s. I thought the concordat’s, I guess they don't want to say alignment, right? Even though the name of the report has alignment in it, but people have different ideas of what alignment means, anyway.
Mike Petrilli: It's a-
Amber: I digress. The bottom line is to determine the alignment between Common Core standards and NAEP, to the reasonable, I'll give you some details in a minute.
Mike Petrilli: Okay.
Amber: At both great levels, especially since NAEP is by design supposed to be broader than Common Core.
Mike Petrilli: Mm-hmm (affirmative).
Amber: It's supposed to, this is a direct quote, "It's supposed to maintain a degree of independence relative to the current fashions in instruction and curriculum." Okay?
Mike Petrilli: Okay.
Amber: Besides that the Common Core writers had access to the NAEP frameworks when they were writing the common core.
Mike Petrilli: Right.
Amber: Right, so okay, anyway. The study results, more specifically, at grade 4 panelists find that 79% of NAEP items were matched to the content that appears in the Common Core at, or below, grade 4, or below, right? Not just at, thinking like below. 79%.
RobertPondiscio: 21% did not?
Amber: That's right. 12% because there's some category where they couldn't make sense of it, or they disagreed, but 12% of NAEP items were judged to assess non Common Core content or were found at Grade 5 or above. The content area with the lowest percentage of NAEP items covering it, that's found in the Common Core at, or below, grade 4. Any idea?
Mike Petrilli: Procedural fluency?
Amber: Nah. Data analysis, statistics, and probability.
Mike Petrilli: Oh.
Amber: Where only 47% of the items were classified as matches. At grade 8 the overall alignment of NAEP to Common Core, at or below grade 8, is 87%. Although there is much variation across the content areas, 7% of NAEP items assess something that's not in the Common Core or it's in the high school. A couple of other little factoids.
There is less agreement between Common Core and NAEP when it comes to algebra and geometry.
Mike Petrilli: Mm-hmm (affirmative).
Amber: Just over half of grade 3 and 4's Common Core's standards, and the Algebra domain, have at least 1 NAEP item match to it.
Mike Petrilli: Mm-hmm (affirmative).
Amber: Okay, so that's. Again the Data analysis, statistics, and probability as we found in grade 4, also had the greatest proportion of items judged to access content not covered in Common Core. Basically you got those. That's your one area where you don't have clear matches.
Mike Petrilli: In other words, there's more of that stuff in the NAEP framework then in the Common Core?
Amber: That's right.
Mike Petrilli: Right.
Amber: Then there's a sort of this weird ending that seemed kind of conflicting to me. On the one hand these authors say that it's been 10 years since NAEP has conducted a thorough review of its math framework. They recommend that we do so in light of Common Core, as well as other college and career ready standards, but I'm thinking didn't we just learn that independence is a really important thing and that we don't want to be caught up in the fad. By the way this alignment is pretty high already, right?
Mike Petrilli: Right, well.
Amber: I'm a little confused.
Mike Petrilli: Well, couple things to add. One is they try to keep the trend line, particularly for the long term trends, right?
Amber: Right.
Mike Petrilli: That pretty much stays standard, but they do tend to update these frameworks every once in awhile to be somewhat matching what kids are learning in schools so that is a tough decision. I did see some folks say today, that the areas where the kids really bombed on the 2015 test, were those that had the least amount of alignment with the Common Core. When you look at the sub scale scores, and get into some of those areas where basically save, under Common Core 4th graders now don't get to those topics until the 5th grade, and they did not do well on them. I don't know. I think it's a very slippery slope to make these arguments when we've held NAEP up as a gold standard. It could be the case that Common Core for good reasons waits until 5th grade, let's say to do some of these things, and the kids just haven't gotten to it yet.
RobertPondiscio: Does this explain some of the potential softness that we are seeing in 2015 data?
Mike Petrilli: Yes, it could. It absolutely.
RobertPondiscio: It means it’s worth learning more about.
Mike Petrilli: Yes, it is a hypothesis.
RobertPondiscio: Right.
Mike Petrilli: We will not claim that be the case because that would be mis-NAEPery 00:17:57].
Amber: We say we don't and then we just do mis-NAEPery 00:18:01] don't we?
Mike Petrilli: No. As long as you state it as a hypothesis Amber, it's not mis-NAEPery.
RobertPondiscio: Conjecture! So now you're a researcher. Go off and tell us what you find out.
Amber: Is this the role of the game?
Mike Petrilli: Yes, it's not a game! Amber.
Amber: Then you look high and mighty because you say that you aren't doing it, but you do it, and then you say but I didn't mean to do it? I don't really get it.
Mike Petrilli: Amber you clearly are missing the nuances here.
RobertPondiscio: She's missing out. She knows, she knows!
Amber: Anyhow. Hey, it was a good study. Honestly I thought it was a little bit more matching going on then I thought.
Mike Petrilli: Yeah, yeah. There is a lot of overlap. Look it is a difficult question. Should NAEP change? I don't know. I don't know. I think there's pros and cons on either side.
Amber: I'm with you.
Mike Petrilli: We do need some kind of benchmark that is not corrupted by the Common Core.
Amber: That's right.
Mike Petrilli: Alright. By the way Amber, I understand we may have some alignment studies coming out.
Amber: Maybe.
Mike Petrilli: As well.
Amber: Maybe this little tiny study that I've been working on for I don't know how long. Yes.
Mike Petrilli: Yes. With MCAS, PARCC, Smarter Balance and ACT aspire coming along.
Amber: Coming soon. Giving me many grey hairs to cover up with blonde dye.
Mike Petrilli: You do that so well. Okay. That is all the time we've got for this week. Until next week!
RobertPondiscio: I'm Robert Pondiscio.
Mike Petrilli: Watch him NAEP, and
RobertPondiscio: Watch me NAEP, NAEP.
Mike Petrilli: Watch him wonk, wonk. Okay. I'm Mike Petrilli. Nae, nae-ing off.
Promises and pitfalls of using NAEP data to assess states’ roles in student achievement
OK, everyone, back away from the ledge. With the release of NAEP data this week, the predictable deluge of commentary is well underway—mainly of the gnashing-of-teeth, rending-of-garments variety. NAEP may be the nation’s report card, but it is also the nation’s Rorschach test. Perception is in the eye of the beholder, and many see darkness and misery: “A Decade of Academic Progress Halts,” says the Los Angeles Times. “Student Score in Reading and Math Drop,” says U.S. News & World Report.
One of the frequent criticisms of NAEP punditry is “misNAEPery”—the sin of attributing fluctuations to particular policies, for example. One particularly virulent form of this fallacy—failure to account demographic changes in states over time—has become slightly less tenable this week, courtesy of this illuminating analysis by Matthew Chingos of the Urban Institute.
Not every state is the same. States with higher concentrations of black and Hispanic children, low-income families, and English language learners (ELLs) have a harder time rising to the top because they have more students mired at the bottom. But when you adjust for these demographic realities, a different NAEP emerges. There’s Massachusetts, still sitting pretty atop the tables. But Texas and Florida, states with higher concentrations of traditionally low-performing subgroups, suddenly look like standouts. Well-regarded high-fliers like New Hampshire, Minnesota, and Vermont begin to look fairly pedestrian, while curve-busting states like Mississippi, Louisiana, and New Mexico—though still low in absolute terms—strongly outperform their peers given where you’d expect states with their particular demographic mix to land.
Perhaps the bigger piece of news, and one that should give cheer in the face of the dismal NAEP results, is this: Between 2003 and 2013, NAEP scores in all fifty states have increased more than would be expected based on demographic shifts over that time period. In other words, with low-income kids and other challenging demographic groups making up an increasing share of the country’s K–12 population, it’s been great and surprising news that NAEP scores have increased. All other things being equal, we should have expected them to go down.
Although the adjusted figures allow for more meaningful comparisons between states, “they are still unlikely to reveal the causal impact of particular education policies,” Chingos warns. “There remain a number of unmeasured student characteristics and differences in non-education policies that could shed further light on these data,” he observes.
The report was published in anticipation of this week’s NAEP release; Chingos has rerun some of the data on the Urban Institute’s website. In all, it’s a fascinating contribution that makes the sin of misNAEPery slightly harder to commit. Or abide.
Now that we’ve seen NAEP through this new and pertinent set of lenses, I’m going to wait before opting to defenestrate. I suggest you do the same.
SOURCE: Matthew Chingos, “Breaking The Curve: Promises And Pitfalls Of Using NAEP Data To Assess The State Role In Student Achievement,” Urban Institute (October 2015).
Common Core's alignment with the 2015 NAEP math assessment
A new study by the NAEP Validity Studies Panel analyzes the alignment of the assessment’s 2015 Math Items (the actual test questions) for grades four and eight to the math Common Core State Standards (CCSS).
To do so, the panel enlisted as reviewers eighteen mathematicians, teachers, math educators, and supervisors who have familiarity with Common Core. This group classified all 150 items in the 2015 NAEP math pool for each grade as either matching a CCSS standard or not.
The reviewers determined that the Common Core and NAEP were reasonably aligned at both grade levels— not surprising, since CCSS writers had the NAEP frameworks at their disposal. Further, NAEP is by design broader than the CCSS and is supposed to maintain a degree of independence relative to the “current fashions in instruction and curriculum.”
Panelists found that 79 percent of NAEP items were matched to the content that appears in the CCSS at or below grade 4. The overall alignment of NAEP to CCSS standards at or below grade eight is even closer, 87 percent.
There is, however, variation in matches across content areas. In fourth grade, the least aligned content area was data analysis, statistics, and probability, in which only 47 percent of the NAEP items matched a Common Core Standard. There’s also little agreement between the CCSS and NAEP when it comes to algebra and geometry. For example, just over half of all third- and fourth-grade CCSS standards in the algebra domain have at least one NAEP item aligned to it.
The authors say that it’s been ten years since NAEP conducted a review of its math framework (a framework that is rather mediocre by our estimation) and recommend that they do so in light of both the Common Core and other college- and career-ready standards. But not so fast. Should we open that can of worms if independence is indeed prized—especially when the alignment between the two is already pretty high?
SOURCE: Phil Daro, Gerunda B. Hughes, and Fran Stancavage, “Study of the Alignment of the 2015 NAEP Mathematics Items at Grades 4 and 8 to the Common Core State Standards (CCSS) for Mathematics,” American Institutes for Research (October 2015).
Public charter schools must become even more innovative
Although the charter sector has grown rapidly in both size and quality in recent years, there are still myriad issues holding it back from substantially improving public education. Most worrisome is the way charters have begun to resemble the district schools they were designed to differ from. In this new paper, the Mind Trust teams up with Public Impact to shine a light on how the sector can embrace its innovative roots in order to improve. The report outlines three key ideas: The sector must get better (slightly edging out traditional public schools isn’t good enough); the sector must get broader (underserved groups like at-risk students, special education students, English language learners, and students in rural communities still aren’t served effectively by charters); and bigger (approximately one million students are currently on charter waiting lists nationwide). The authors emphasize that creative thinking and innovation are the only ways forward in accomplishing these goals. Trying the same old things on new student groups, working harder instead of smarter, and failing to find more effective and sustainable ways to operate won’t expand the impact of charters. Instead, they will only deepen their similarity to traditional schools.
To achieve break-the-mold results, stakeholders will need break-the-mold innovations. The report offers recommendations and parses them out according to four stakeholder categories: school operators, policy makers, city-based education organizations, and funders. Perhaps unsurprisingly, the recommendations for school operators are the most detailed. The authors advise them to invest in new school models that innovate in the realm of student experience, as well as the use of time, space, and personnel. These new models could come from start-up organizations, existing high-flyers, or collaboration between the two. Operators can also embrace innovation by making ambitious improvements to existing models. The report recommends adjustments to facility use and facility funding; investing in taking over failing schools rather than planting new ones; rethinking startup economics by opening entire schools at once rather than phasing in by grade; backfilling vacant spots; and sharing school functions like administrative and finance work across networks.
For policy makers, supporting innovation includes funding charters equitably, permitting high-performers to expand, and setting (and acting on) high performance expectations. City-based education organizations can advocate for policies that support innovation; facilitate connections between new models, existing schools, and local leaders; offer planning funds; attract talent; and support small-scale pilots. For funders, the recommendation is simple and straightforward: Invest in new and innovative models, even if it means taking a risk.
The risk factor, of course, is what will ultimately determine whether stakeholders heed this report’s mostly solid recommendations. Considering that charters were founded because of a need for something different from traditional schools, one can only hope that stakeholders show greater willingness to innovate. Otherwise, we may end up with a pair of traditional—and struggling—school systems.
SOURCE: “Raising the bar: Why public charter schools must become even more innovative,” The Mind Trust and Public Impact (October 2015).