Former California state superintendent Bill Honig recently wrote a blog post criticizing the recent Fordham study that we coauthored, Rating the Ratings: An Analysis of the 51 ESSA Accountability Plans. Although we respect his opinion, we take issue with two of his arguments.
First and foremost, Honig accuses us of incorrectly assessing California’s approach to measuring student achievement, and claims that our alleged mistake was the result of “sloppy staff work.”
We are happy to admit when we’ve erred, as with our inaccurate analysis of New Hampshire’s plan, but in this case we beg to differ.
An important part of our process was corresponding with each state’s education department to ensure that our ratings were correct. We did this in every instance, including California. That’s for a sound reason: State ESSA plans are often opaque and confusing, and we wanted to be careful not to mischaracterize them. Yet when we shared our draft review with the California Department of Education, all we got back was an angry note about our “superficial and inadequate process.” We requested, yet again, confirmation that we were accurately describing California’s accountability system, but we received no further response.
In particular, Honig thinks we should have given California credit for looking at student achievement at more than the proficient level. That the state does this, though, is hardly obvious from the plan. The most relevant section of California’s document says, “Proficiency is measured by looking at a student’s Distance from Level 3 (at each grade level), which compares how far above or below students are from the possible scale score to achieve Level 3 (Standard Met) on the administration of the Smart Balanced Assessments. Currently, ‘Status’ [the state’s term for achievement, as opposed to growth] is determined using the average of these distances...”
Clear as day, right?
Yes, perhaps this could be interpreted to mean that California is using a performance index of some kind that gives students different amount of credit for achieving at, for example, “basic,” “proficient,” and “advanced” levels, but that’s far from the plain meaning of that sentence. That sentence has no plain meaning. It could as easily mean that the state is using this method to determine whether each student is or isn’t proficient, and nothing more. This ambiguity is why we asked for California’s confirmation that we got it right.
Nevertheless, if the California Department of Education contacts us and explains why the state should have received credit for the way it measures student achievement, we will of course amend our report. Until then, we cannot confidently find in those words the assurance we would need before accepting Honig’s contention.
We also take issue with his critique of our call for states to make student growth count for half of school ratings. He claims:
Fordham’s fix is terribly flawed. They want growth in student test scores in math and reading to be at least fifty percent of the total grade of the school. If growth is emphasized to the exclusion of status (the actual performance level), then schools which have historically produced students scoring at high levels are mistakenly identified as mediocre or worse. Imagine a school with low-income students who after considerable effort has reached a high plateau of performance and maintained that level for several years. They would unfairly look mediocre or worse on a measure heavily weighted to growth.
There are a number of problems with this paragraph. For starters, we want growth of any kind—not just math and reading—to count for 50 percent of a school’s total grade. We state this clearly multiple times in the report. And his more substantive point of contention is misguided at best. Is he suggesting that there are schools serving grades K–8 in which students have already maxed out their knowledge of subject matter? And what “plateau”? A student entering any grade can—and obviously should—learn more in that grade, and an assessment can and should measure that growth, which is the growth for which we’re advocating. Yes, it’s possible for a student to hit a “ceiling” on a deficient test if they are super high achieving, but California uses Smarter Balanced, a computer-adaptive test that is designed to avoid that problem in all but the most extreme cases. So what’s Honig’s worry?
We have a few more beefs with Honig’s review, though they tend to be more of the honest-disagreement variety. We don’t think an elaborate California-style data dashboard, with color coded ratings against dozens of indicators, will give parents and the public a clear understanding of whether a school is excellent, terrible, or somewhere in between. We see California’s approach as a dodge, and not worthy of being labeled an accountability system.
California’s officials, past and present, are of course free to disagree with the assumptions and principles that drove our analysis. What we can’t allow them to do is disparage the quality of our work, especially when we took every effort to make sure we got it right and our error—if indeed there is one—is properly laid at the door of the California Department of Education.
