It's not easy today even to recall the stir created in 1987 when an obscure West Virginia physician and his never-heard-from-before one-man advocacy organization called "Friends for Education" released a little study titled "Nationally Normed Elementary Achievement Testing in America's Public Schools: How All 50 States Are Above the National Average." Swiftly dubbed the "Lake Wobegon report" after Garrison Keillor's mythical Minnesota hometown where "all the women are strong, all the men are good-looking, and all the children are above average," it rocked the testing and education policy worlds, which were still reeling from A Nation at Risk just four years earlier.
As Education Week's Bob Rothman reported at the time, "The overwhelming majority of elementary-school pupils nationwide score above the average on normed achievement tests, results from a controversial new 50-state survey indicate. The survey...provides average scores from the 32 states that test elementary students statewide, as well as selected averages for district-administered tests in the 18 states without statewide assessments... Such findings suggest that norm-referenced tests - in which students are compared with a group tested in the past, not with other current test takers - 'do not represent an accurate appraisal of educational performance,' argued John Jacob Cannell, the group's president and author of the report.... Based on the most recent results available, 90 percent of school districts and 70 percent of students tested performed above the average on nationally normed tests, the report estimates. Those scores are inflated because testing companies set norms that are artificially low, Dr. Cannell charged. Test publishers 'want to have good news to sell to superintendents,' he said in an interview."
It was a bombshell precisely because nothing like it had been said before and because Americans were accustomed to trusting their local superintendents and newspapers to supply accurate information about school performance. Though the Education Department's controversial "Wall Chart" annually compared state academic achievement using such measures as SAT and ACT scores, not until a year after Cannell's report did Congress agree to permit NAEP results to be reported state-by-state (and that practice remained optional for states until NCLB was enacted in 2001). International assessments were rare and their results reported in such arcane ways as to make comparison essentially impossible. Since states and districts used tests of their own choosing and reported their own results as they saw fit, other than national NAEP results Americans had no practical way of gauging the achievement of children in one state versus another or in relation to the country as a whole. More important, there was no effective external audit by which state and district test score reports could be appraised for their honesty and accuracy.
Cannell's blast altered that landscape. We already knew the country was "at risk" but nobody supposed that most Americans were getting misleading information about their own schools' performance. Cannell suggested, in fact, that most people were living in an education fool's paradise, duped by the complicity of school superintendents and testing companies to report cheery news even when unwarranted by the facts. His report began to explain that familiar yet puzzling Gallup finding wherein people conferred low grades on the education system in general but high marks on their own kids' schools.
Why recall this tale eighteen years later? Because NCLB still allows states to select their own tests, to define "proficiency" however they like, and to choose their own passing scores on those tests. And because Paul Peterson's and Rick Hess's recent analysis of state proficiency claims versus NAEP results suggests that the majority of states are still awash in unwarranted good news about their students' true performance (see here). Of the forty states that Peterson and Hess were able to compare, just five earned "A" for defining and measuring proficiency on their state test in ways that yielded results akin to NAEP's. The average gap between a state's proficiency claims as gauged on its own tests and the fraction of its kids deemed proficient by NAEP was a staggering 36 percent (in fourth grade reading). That means if one accepts NAEP's standard of proficiency as a suitable benchmark, the overwhelming majority of states are categorizing millions of youngsters as proficient who really aren't. Most of the country, in other words, is still paddling in Lake Wobegon.
Though the testing industry and many state and local officials rose up to smite Dr. Cannell back in 1988, Bill Bennett and I, then at the Education Department, asked the federally funded research center on testing and measurement, based at UCLA, to examine his findings. Their conclusion: Cannell may have exaggerated a bit but he was essentially correct. Said testing expert Dan Koretz, "He was clearly right about his basic conclusion. With respect to national averages, districts and states are presenting inflated results and misleading the public."
To be sure, Cannell is a bit of an eccentric. He vanished from sight soon after his fifteen minutes of testing fame; moved to New Mexico to study psychiatry, then forensic psychiatry; and is now on the staff of California's largest hospital for the criminally insane. In his spare time, he directs the "Vitamin D Council," which contends that "many humans are needlessly suffering and dying from Vitamin D deficiency" (See here).
But he hasn't lost his activist's passion, his crusader's zeal or his interest in honest score reporting. Recently, he sent me (and Margaret Spellings, Arnold Schwarzenegger, etc.) a letter noting that No Child Left Behind "does not place any restraints on state officials telling large numbers of students they are testing above the 'national average.' Nor does it prevent school officials from cheating; in fact, it encourages cheating by greatly raising the testing stakes." The solution, he contends, is a single, secure, NAEP-style "national achievement test."
And you know, he might just be as right about that in 2005 as he turned out to be about test-score reporting in 1987.
