Dr. Pangloss, the incorrigible professor of "metaphysico-theologico-cosmolonigology" in Voltaire's Candide, insisted, despite all evidence to the contrary, that he was living in the "best of all possible worlds." It was with this propensity for self-delusion in mind that Education Sector named its annual study on how states game No Child Left Behind (NCLB) the "Pangloss Index."
The study ranks the states based on annual progress reports that they themselves submit to the federal government under the terms of NCLB. Because the states define most of the terms and criteria in these reports themselves--things like "proficiency" and "graduation rate"--their scores are more reflective of how they rank their own schools than how they stack up against some objective standard.
Some jurisdictions clearly take seriously their reporting responsibilities. The numbers provided by the District of Columbia, for instance, land it at the very bottom of the list. That sounds about right. Then there's Massachusetts, which has such tough standards that it ranks 46th on the Pangloss Index-despite having the country's top NAEP scores in both 4th and 8th grade.
Some states, however, have their heads even deeper in the sand than the professor himself. The report heaps particular scorn on Alabama, which abandoned all sense of reality when the numbers it reported in 2007 caused it to jump 17 ranks from the 2006 Pangloss Index. The state now sits at fifth place (even though its 2007 NAEP scores are near the bottom). As the report makes clear, officials in the Heart of Dixie adopted a smoke-and-mirrors strategy that explains this dramatic "progress" over time.
Here's a short (non-exhaustive) list of some of the "accountability-avoidance gambits" that Alabama has cooked up in the past 5 years:
- Established a minimum subgroup size of 40, higher than in 31 states, which allowed it to exclude a large proportion of its disadvantaged and minority students from AYP calculations.
- Set cut scores extremely low on state tests. In 2003-04, 76 percent of 4th graders scored "proficient" on the state test, while only 23 percent did so on NAEP.
- Used "confidence intervals" (like the "plus or minus" disclaimer used in popular polls) to artificially boost proficiency rates.
- Adopted a "performance index" whereby half the students who reached the laughable "basic" level on the state exam could be counted as "proficient" for AYP purposes.
The obvious question, of course, is how do states like Alabama (and there are, in this regard, a lot of states like Alabama) get away with such duplicity? As the report makes clear, the Department of Education mostly rubber-stamped Alabama's (and most other states') ridiculous requests. It was only when an Associated Press story on the "minimum subgroup size" ruse stirred Congressman George Miller to wax militant on NCLB compliance that the Department finally turned down a proposal from Alabama officials. (They wanted to define "minimum subgroup size" as the greater of 40 students or 10 percent of the school population--which in a thousand-student school would, obviously, be 100.)
Education Sector properly urges Congress to close these and other loopholes and to leave less room for interpretation (and manipulation) in future iterations of NCLB. (They'll have a while to ruminate on this, as reauthorization looks dead until at least 2009.) The experience of Alabama, say the authors, should serve as "a cautionary tale for members of Congress working to write the next version of the nation's most important education law."
Unfortunately, due to a combination of limited foresight and inevitable political compromise, lawmakers just aren't very good at closing loopholes. And government agencies, such as the Department of Education, are even worse at it. Congress needs to put down the pens and pull out the scissors.
They should start with NCLB's "100 percent proficient by 2014" mandate, which forces states either to find all of their schools to be failing (which is dumb policy and dilutes accountability) or to play the very games described above. They should also relieve states of the temptation to dumb down their tests by putting in place a single exam. The best way of doing this is debatable (Mandate a national test? Convene a panel of governors? Piggy-back on the American Diploma Project or the Advanced Placement program?), but its value is clear: no more governors telling kids who can't read, write, add, or subtract that they're making "tremendous progress," as Alabama Governor Bob Riley did earlier this year.
Of course, a national standard won't fix everything. And who's to say the ever-accommodating Department of Education would even bark, let alone bite, when states start gaming the next version of NCLB?
