Margins of Error: The Education Testing Industry in the No Child Left Behind Era
Thomas TochEducation SectorJanuary 2006
Thomas TochEducation SectorJanuary 2006
Thomas Toch
Education Sector
January 2006
In Education Sector's inaugural publication, Thomas Toch presents a compelling account of NCLB's impact on the testing industry. His bottom line is that the "complex test-making infrastructure is buckling under the weight of NCLB's testing demands." As testing companies expand rapidly, reports Toch, test quality is being left behind. Psychometricians jump ship from state departments of education to more lucrative jobs in the testing companies, and states then lack the brainpower and muscle to keep those companies on the straight and narrow. The result? An increasing rate of blatant scoring mistakes that has caused students to, for example, miss graduation (because of supposedly failing test results), and forced others to attend summer school. But there are subtler problems as well. Because of the prohibitive cost, testing companies resist developing exams that probe beyond basic skills. And unless states are willing to advocate and pay for more complex assessments, the companies are unlikely to provide them. None of this spells a major affront to standards-based reform or NCLB; Education Sector is careful to communicate its strong support for the law. But it would have the feds play a stronger role in improving the country's testing infrastructure, at the least by paying for the training of 1,000 new testing experts and the development of stronger state tests. Its boldest recommendation is for the development of national tests. Beyond the myriad contributions such tests could make to education policy and practice (see here and here), they would also be more cost-efficient than the 50-state patchwork we have now. With a serious discussion of national standards and tests breaking out across the ideological spectrum, is there reason to hope that the time is right? We think so-but we also remember who worked behind the scenes (and put up big money) to torpedo President Clinton's "voluntary national tests" back in the 1990s: the selfsame testing industry. Now it has even more money-and more to lose. You can read the Education Sector report here.
Fritz Edelstein
The United States Conference of Mayors
2006
This slick but welcome 35-page booklet seeks to "assist mayors in making choices in how they may get involved in the school systems of their cities." But it broadly construes "school systems" and "getting involved": the former includes such alternatives as charter schools, and the latter such bold strategies as mayors assuming "total control" of school systems. It does not, however, push mayors to take any particular approach to K-12 education. Instead, this primer spells out reform options and considerations. It offers a four-course menu of strategies-including vanilla versions, too, such as "partnership" and "medium involvement"-and supplies information about cities where each strategy is currently in practice. Three issues singled out for special attention are the relationship between the mayor's office and the school system "central office"; the urgency of understanding school budgets; and the concept of creating a "portfolio" of schools. If the last of these sounds familiar (as well as worthy), it may be because the Gates Foundation underwrote the project. You can find it here.
The New York Times editorial page, pulling from a story by one of its own reporters, takes on NCLB's free tutoring provision, but swings and misses. It rightly expresses concern that only 12 percent of the eligible students nationwide are receiving tutoring services. But it excuses the primary culprits: most of the big urban districts. Pittsburgh, for example had 3,000 eligible students last year and the funds to serve 2,000-but only served 100. On the other hand, Baltimore embraced tutoring and provided it to 4,000 students-about 80 percent of what the district could afford with the federal dollars allotted. This contrast-certain urban districts meeting their responsibilities while others shirk them-is the real story. Where's the outrage that some districts are doing all they can to keep parents in the dark about the free tutoring, since they retain the money if kids don't sign up? Can you imagine the Times' reaction if districts were keeping the free lunch program a secret from poor families?
"Tutoring Gap," New York Times, February 16, 2006
"Tutor Program Offered by Law is Going Unused," by Susan Saulny, New York Times, February 12, 2006
For Jenna Helenski, the daily grind at Connecticut's Ellington High School can be exhausting. Thank goodness the senior has three study halls everyday to catch her breath. And she isn't alone: more than half of the school's seniors have three daily study halls-during which eating, chatting, or iPod listening are de rigueur- and the vast majority of all students at Ellington have at least two. "I wouldn't change it," said Helenski, referring to the current Ellington system. "You need the breakup, the time to recover between classes." Students' balanced chakras aside, some parents have complained that their kids are wasting time in the cafeteria instead of learning in the classroom. The study halls, Ellington's principal replies, are simply a result of the district's lack of funds and its inability to hire more teachers for elective courses. (So who's paying the hall monitors?) Regardless, Gadfly wonders why administrators would warehouse kids in the lunchroom instead of encouraging them to, say, take classes at a local junior college. Perhaps some seniors could spend part of their day in an internship or job-training program? Or maybe the state should follow Ohio's example (see Education tasting menu, above) and think about beefing up its graduation requirements. Anything, really, but what the school is doing now.
"Too Many Students Stuck In Study Halls," by Rachana Rathi, Hartford Courant, February 12, 2006
Tonight, Chef Finn has prepared four courses. Bon Appetit!
The "65 Percent Solution"
A well-meaning young business entrepreneur has started a well-meaning organization called "First Class Education," (http://www.firstclasseducation.org/) whose sole mission is to get states to enact laws requiring schools to spend at least 65 percent of their "operating budgets" in the classroom. That straightforward idea appeals to such thoughtful critics as George Will and, now, to such reform-minded states as Arizona, Florida, Georgia, and Texas. Advocates claim that many billions of dollars now spent on various school overheads and low-priority activities would be spent on actual student instruction if this policy were put into place.
Well-meaning, to be sure, and consistent with the oft-noted fact that school systems spend vast sums on lower-priority activities than teaching and learning. But, like most formulaic solutions, it's too simple-and apt to retard other important reforms that K-12 education also needs.
An example of how it's too simple: School libraries and librarians aren't counted as "classroom" expenditures. Yet field trips are.
An example of how it may hold back other valuable reforms: distance-learning offers enormous potential to bring high-quality instruction, and instructional materials, into student's lives, whether in the classroom, at home, or at the local Girls & Boys Club. For some kids, it's the best way. Yet the technology revolution, the capital investments, and the salaries of gifted instructors and curriculum developers on the other side of the nation or planet presumably don't count as expenditures "in the classroom." To shackle a state's or school system's education budget to such a formula may serve to freeze the status quo and deter far more powerful means of educating children.
Simple external controls have both the virtues and the shortcomings of simplicity. Remember wage and price controls as means of curbing inflation? It turned out that what had to be done was to solve the underlying economic problems. Same with schooling.
Why Not the Obvious Solution to Teacher Shortages?
State after state and city after city are lamenting their shortages of teachers in general, math and science teachers in particular, and "highly qualified" teachers above all. The Maryland Department of Education reported this month that just 42 percent of Baltimore teachers meet the federal definition of "highly qualified," even as California education analysts predict a shortage of 100,000 teachers in that state over the next decade. Why, for Pete's sake, are they not pursuing the obvious remedies, namely differential pay, alternative certification, slashed red tape, portable pensions, and all the rest? Such measures were even recommended in Maryland by Governor Ehrlich's 2005 commission, but nobody there has lifted a finger to implement them. Of course, we know why not. Adult vested interests-teacher unions, ed schools, etc.-don't want such reforms, and (as the California Teachers Association showed Governor Schwarzenegger last November) they'll fight hard and spend lavishly to sustain the status quo. In Maryland, Ehrlich is running for re-election and, at least in the education sphere, he's lavishing money on that selfsame status quo and rocking no boats. Pity.
Three Cheers for Taft
Ohio's beleaguered GOP governor started his final year in office with a "state of the state" address that contains at least one really good idea-his "Ohio Core" initiative. In Taft's words:
"One in four freshmen don't [sic] come back for a second year. Forty percent of college freshmen will never earn a degree. For too many, a high school diploma is not a passport to success, but rather a broken promise....We know that Ohio students who take a rigorous core curriculum are less likely to require remedial course work and are more likely to succeed on the job, or in college.... Here's the plan:
"First, require all students to take rigorous course work that will prepare them for the workforce or college-this means four years of math, including Algebra II; three years of science, including biology, chemistry and physics; four years of English; three years of social studies; and at least two years of a foreign language. To give families and schools time to prepare, the core curriculum should apply to students in the graduating class of 2011.
"Second, make completing a rigorous core curriculum a condition of admission to Ohio's state-funded four-year colleges and universities.
"Third, move all remedial education to Ohio's two-year campuses, where costs are lower.
"Fourth, require all students to take a college and work-ready assessment in their junior year to help them know if they're on the right course to be prepared for life after high school.
"Finally, add a measure to the School Report Card to indicate how well high schools are preparing students for college and work.
"We must also help students earn a college degree more rapidly at lower cost. We should give every high school student in good standing the opportunity to earn at least one semester of college credit while still in high school."
Can Taft deliver? The K-12 system will balk at raising standards further when so many young people are already failing the Ohio Graduation Test; the state's famously standoffish university system may welcome applicants who have completed Taft's proposed "core," but that doesn't mean they'll necessarily be accepted-or admitted into credit-bearing courses; and the community colleges are already gagging at being cast as "remedial" institutions. Still, in the spirit of the American Diploma Project, Taft is nudging the Buckeye State in the right direction. And he's not the only one working to give high school standards a boost. Florida Governor Jeb Bush's new A++ plan pushes for tougher core curriculum study in the Sunshine State (see here).
And since getting there will also require more and better high school science and math teachers, what about a major move toward alternative certification and lowered entry barriers for Ohio, too? (For additional information about the Taft plan, see here.)
The Limits of Achievement Comparisons
The New York Times seems fixated on publicizing bad studies that paint traditional public schools in a good light and charter/private schools as failures. Since the paper's editors aren't dumb, it must be ideological. The latest outrage was Diana Jean Schemo's effort to lend credibility to a miserable piece of work by the self-styled "National Center for the Study of Privatization in Education" at Columbia University's Teachers College. Inexplicably funded by the National Center for Education Statistics-thus giving it the cachet of a "federal study"-this analysis, like all achievement analyses based on NAEP data, can tell you about student achievement differences but not about whether the schools caused them. Schemo misleadingly quotes Howard Nelson of the American Federation of Teachers: "studies seem to show that charter schools do no better, and private schools do worse." But these are "snapshot" data, not longitudinal or value-added data, and the most one can do with them is say that "students attending X schools score higher/lower than students attending Y schools." Analysts know nothing about where those kids were when they entered the school, how long they've been there, or what gains they may or may not have made while enrolled there.
As for efforts to "control" for student demographics and suchlike, that's a battleground of methodological disputation. Anyone sophisticated at this sort of analysis is able to devise "controls" and "adjustments" that can pretty much make data jump through whichever hoops one favors. (One could control, for example, for the race and income of patients in maternity hospitals versus cancer hospitals, yet still have radically discrepant death rates based on inherent differences in the severity of their medical problems upon entry.) I do not claim (or believe) that private schools are inherently more effective than public. I've sensed for ages that their achievement edge may have more to do with their admissions policies and the kids' families. But my parallel observation of charter schools also indicates that the relatively low achievement of some of their students owes much to the awful experiences those youngsters had in their previous district public schools, before entering the charter world. Snapshot analyses can tell you none of this. Shame on the Times for pretending otherwise.
A self-avowed creationist booted intelligent design from Ohio's science program on Tuesday. Martha Wise, the member of the state board of education who asked her colleagues to strike the "Critical Analysis of Evolution" model lesson plan from state curriculum materials, celebrated her victory by saying that this decision is "good for students...science...and the state of Ohio." She tried the same strategy last month, but lost by a 9-8 vote. Perhaps governor Taft's threat to send the disputed lesson plan for legal review in the wake of the recent federal court decision finding that intelligent design wasn't science (see here) catalyzed the board's change of heart. Wise has taken heat from the intelligent design crowd, who ask how she could do this to them. "I can do [it] because I believe there are two separate issues here. One is the teaching of good science. The other is the teaching of creationism...that should be taught...at church or at home...not in science class." Hallelujah.
"Darwin Prevails in Latest Skirmish," by Carrie Spencer Ghose, Cincinnati Post, February 15, 2006
"Challengers of Evolution Lose," by Lisa Anderson, Chicago Tribune, February 15, 2006
E-Comp-Florida's newly-announced performance pay plan that would give bonuses to the top 10 percent of the state's teachers-is an ambitious and promising reform proposal.
Sunshine State law says that teachers ought to be rewarded when their students are learning. Under the plan announced last week by the state Department of Education, the one-teacher-in-ten whose classes show the greatest improvement on the Florida Comprehensive Assessment Test (FCAT) will earn a 5 percent pay bonus, which on average would amount to approximately $2,000. Another part of the plan bases a portion of all teachers' pay on their students' achievement-which can accumulate over time to sizable rewards for effective instructors. Teachers won't have to fill out applications or jump through any hoops-the state will do the calculations, based on the Florida FCAT assessment system.
Unfortunately, E-Comp is not receiving the serious discussion it deserves. Instead, it's receiving predictable brickbats from the usual suspects. Even a lawsuit.
One gripe is that E-Comp supersedes district decision-making. In fact, the state had tried since 2002 (when Florida's performance pay law was passed) to leave bonus distribution in district hands. But in most districts that worked horribly. To receive their bonuses, teachers had to navigate mazes of paperwork and complicated applications. Some districts constructed so many bureaucratic barriers that after three years they've still not awarded a single bonus. Last year, 24 districts (more than a third of all those in the state) did not award any performance-pay money.
The legislature's intent was plainly being thwarted. With E-Comp, the state abolishes convoluted application procedures and simplifies the bonus criteria so that deserving teachers will automatically receive the stipends to which they're entitled.
Much of the resistance to E-Comp is due to its reliance on the FCAT. But the FCAT is the cornerstone of Florida's accountability system (which has been responsible for the Sunshine State's progress on national academic measures), and it's the sole method by which the state measures student performance. Districts and schools are already judged (and rewarded or sanctioned) by students' FCAT scores-why not teachers, too?
There are, however, areas where E-Comp is problematic, notably its ambiguity for instructors who teach "non-FCAT" subjects (e.g., art, history, music) and how they will qualify for bonuses. Here the state has charged districts with the responsibility of creating their own assessment mechanisms. Causing one to wonder: If districts bungled the management of performance-based bonuses before, are they up to this responsibility now? (Presumably the state doesn't want to test kids in more subjects than it already is.)
On balance, E-Comp deserves honest debate-and considerable applause. It holds much promise for Florida's students, and for the future improvement of the state's education system.
"State may tie teacher bonuses to tests," by Matthew I. Pinzur, Miami Herald, February 11, 2006
"The FCAT and teacher pay," St. Petersburg Times, February 14, 2006
"Reward Teachers With Merit Pay," Tampa Tribune, February 12, 2006
Is making preschool universally available a good idea? Governor Rod Blagojevich of Illinois thinks so. His proposal to offer universal preschool to all 3- and 4- year-olds in the Land of Lincoln would be the first of its kind in the nation (several other states have universal preschool for 4-year-olds), and he's adamant about seeing the legislature pass it. "We will roll up our sleeves and we will fight," Blagojevich said. And Illinois is not alone. In June, Californians will vote on Proposition 82, which would offer 3 hours of preschool each day to its 4-year-olds by 2010. Virginia's Governor Tim Kaine is also pushing universal preschool through his Strong Start Initiative. While there's little doubt that high-quality preschool programs can benefit youngsters-especially underprivileged and minority students- how do states intend to ensure consistent quality once these programs are brought to scale? Universal preschool is a good idea, but only if it is voluntary, cognitive (i.e., not supervised playtime), and allows parents to choose among providers. If Illinois, Virginia, and California can meet those necessary conditions-while maintaining high quality instruction-then more power to them.
"Doubts cast on preschool proposal,"by Diane Rado and Rick Pearson, Chicago Tribune, February 13, 2006
"Kaine Preschool Initiative To Get State Study,"by Charlie Jackson, Leesburg Today, February 15, 2006
"Should California pay for preschool?," By Dana Hull, San Jose Mercury News, February 12, 2006
Fritz Edelstein
The United States Conference of Mayors
2006
This slick but welcome 35-page booklet seeks to "assist mayors in making choices in how they may get involved in the school systems of their cities." But it broadly construes "school systems" and "getting involved": the former includes such alternatives as charter schools, and the latter such bold strategies as mayors assuming "total control" of school systems. It does not, however, push mayors to take any particular approach to K-12 education. Instead, this primer spells out reform options and considerations. It offers a four-course menu of strategies-including vanilla versions, too, such as "partnership" and "medium involvement"-and supplies information about cities where each strategy is currently in practice. Three issues singled out for special attention are the relationship between the mayor's office and the school system "central office"; the urgency of understanding school budgets; and the concept of creating a "portfolio" of schools. If the last of these sounds familiar (as well as worthy), it may be because the Gates Foundation underwrote the project. You can find it here.
Thomas Toch
Education Sector
January 2006
In Education Sector's inaugural publication, Thomas Toch presents a compelling account of NCLB's impact on the testing industry. His bottom line is that the "complex test-making infrastructure is buckling under the weight of NCLB's testing demands." As testing companies expand rapidly, reports Toch, test quality is being left behind. Psychometricians jump ship from state departments of education to more lucrative jobs in the testing companies, and states then lack the brainpower and muscle to keep those companies on the straight and narrow. The result? An increasing rate of blatant scoring mistakes that has caused students to, for example, miss graduation (because of supposedly failing test results), and forced others to attend summer school. But there are subtler problems as well. Because of the prohibitive cost, testing companies resist developing exams that probe beyond basic skills. And unless states are willing to advocate and pay for more complex assessments, the companies are unlikely to provide them. None of this spells a major affront to standards-based reform or NCLB; Education Sector is careful to communicate its strong support for the law. But it would have the feds play a stronger role in improving the country's testing infrastructure, at the least by paying for the training of 1,000 new testing experts and the development of stronger state tests. Its boldest recommendation is for the development of national tests. Beyond the myriad contributions such tests could make to education policy and practice (see here and here), they would also be more cost-efficient than the 50-state patchwork we have now. With a serious discussion of national standards and tests breaking out across the ideological spectrum, is there reason to hope that the time is right? We think so-but we also remember who worked behind the scenes (and put up big money) to torpedo President Clinton's "voluntary national tests" back in the 1990s: the selfsame testing industry. Now it has even more money-and more to lose. You can read the Education Sector report here.