Evaluating Value-Added Models for Teacher Accountability
RAND Corporation2003
RAND Corporation2003
RAND Corporation2003
RAND-Education is the source of this Carnegie-funded 160-page monograph by Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz and Laura S. Hamilton. Given today's keen interest in value-added evaluations of teachers and schools, you'll want to know about it, even though it's boring and academic and concludes that more research is needed. Still, it does a nice job of setting forth possible sources of error in value-added teacher evaluations and analyzing well-known methods for doing this (including the celebrated methodology of Bill Sanders). In the end, sounding like a (juiceless) version of Churchill discussing democracy, the authors conclude that, notwithstanding the many perils of value-added analysis, it may be no worse than "other methods for using test scores to evaluate schools or teachers" and might even "provide less-biased and more-precise assessments of teacher effects." See for yourself at http://www.rand.org/publications/MG/MG158/.
Andrew J. Rotherham, Progressive Policy InstituteMarch 2004
In this policy report, Andrew Rotherham, director of the 21st Century Schools Project at the Progressive Policy Institute, argues for incentives to lure National Board Certified Teachers (NBCTs) to teach in struggling schools and work with struggling students. NBCTs, he contends, "are disproportionately working in more affluent schools that are less likely to be struggling or having trouble meeting NCLB's teacher quality mandate." In response, Rotherham suggests a sharp increase in the bonus and pay differential for NBCTs and linking these incentives and rewards to state efforts to "help hard-to-staff schools meet NCLB's highly qualified teacher mandate or otherwise help struggling schools improve." Of course these recommendations are only as good as National Board Certification itself, and more research is needed to determine how well this certification identifies effective teachers, especially those who are most likely to be successful in needy schools. (See http://www.edexcellence.net/gadfly/issue.cfm?issue=19#201 for our review of the most recent study of NBPTS certification.) To see for yourself, go to http://www.ppionline.org/ppi_ci.cfm?knlgAreaID=110&subsecid=135&contentid=252498.
Mass Insight Education
April 2004
Mass Insight Education has released a "bicoastal" poll that shows the public in two states wants more math in K-12 education - and has less "math phobia" than most people would believe. The poll, conducted by MIE and the Seattle-based Partnership for Learning, a business group, polled the public on attitudes toward math education in the Massachusetts and Washington State, and found that 75 percent believe that all students should take algebra and geometry, while one-third think all students should take trigonometry and calculus. The poll also looked at the attitudes of business leaders in both states, who say overwhelmingly that students' math skills aren't up to snuff, especially for work in high-tech industries. The public also likes standards-based accountability measures and thinks they've helped to raise student learning. The poll doesn't do much that the American Diploma Project (http://www.achieve.org/achieve.nsf/AmericanDiplomaProject?OpenForm) hasn't done in more detail. But it's a useful temperature-taking exercise and adds additional evidence to support that project's central contention: that a high school diploma is not the useful marker for skills and knowledge that it used to be. You can read the full report at www.massinsight.org.
Brian Anderson, editor of the estimable quarterly City Journal, argues here that tax credits for private school tuitions have a brighter political future than vouchers. He's also sensitive to new roadblocks placed in the way of vouchers by the Supreme Court's ruling in Locke v. Davey, wherein the Court allowed a college scholarship program to be denied to a divinity student attending an evangelical college (see http://www.edexcellence.net/gadfly/issue.cfm?issue=137#1689). Though the Court sought to limit its ruling to the case at hand, not to venture into the murkier depths of indirect state aid to students attending religious institutions, the extension of the argument to vouchers is obvious, and Anderson thinks tax credits may be the better way to advance choice, especially considering the number of states with "Blaine amendments." Worth a try, of course, but will tax credits ultimately prove the easier sell? They're more complicated to use and to explain. They may or may not benefit families with very low incomes. And those who oppose school choice (especially the private-school kind) aren't going to roll over just because the mechanism is the tax code rather than a more traditional "spending" program.
"If not vouchers?" by Brian C. Anderson, City Journal, Spring 2004
In 1993, the Campaign for Fiscal Equity (CFE) in New York City filed a lawsuit against the state, claiming that New York State had failed to live up to its constitutional obligation to provide a "sound, basic education" to all its students. Eight years of litigation later, State Supreme Court Judge Leland DeGrasse decided in favor of CFE, ruling that Gotham students were not being given a sound, basic education because the state was not providing the city adequate educational funding. Last year the New York Court of Appeals upheld DeGrasse's decision, warning, as the New York Post put it, that the state "had better find some way to solve the money problem, fast." Of course, left unanswered was the question of how much money would be needed for "adequate" funding in New York City (and elsewhere). As it is, New York City spends more per-pupil than most U.S. school systems. And, despite his January 2003 rejection of, as Sol Stern characterizes it, the "popular but misguided ideal that inadequate funding was the reason for the miseducation of the city's school children," Mayor Bloomberg and Schools Chancellor Joel Klein are now siding with CFE, which recently called for the state to provide "$8.9 billion in construction aid to the city over the next five years....[and] $5.3 billion in additional operating aid to city schools." (One wonders where this additional $14 billion will come from when the state is already grappling with a $5 billion budget shortfall.) On Tuesday, in fact, Klein delivered an ultimatum to state lawmakers: Give city schools billions of dollars in additional aid or "I'll see you in court." It's a shame that neither CFE, nor the judges who heard the case, nor the mayor and schools chancellor, heeded the evidence presented by experts like economist Eric Hanushek, who argue convincingly that more money makes education more expensive, not more effective.
"New York's fiscal equity follies," by Sol Stern, City Journal, Spring 2004
"The school-spending racket," New York Post, April 14, 2004
"Klein tells Albany: Pay up or I'll sue," by Carl Campanile, New York Post, April 14, 2004
Most people can agree on two propositions: that programs for gifted youngsters are a good thing, challenging the fast learner more than the standard curriculum, so long as they have high standards and expectations for participating students. And that such programs should be readily accessible to students of every race. But what happens when those two principles come into conflict? Writing in the Wall Street Journal, Pulitzer Prize winning reporter Daniel Golden reports that the laudable desire to increase the number of minority (black and Hispanic, not Asian) students in gifted classes is leading schools to relax the entrance requirements and standards of gifted and talented programs. In some states, students who don't make the cut-offs for such programs can still enter if they pass an easier "alternative test" (sometimes tests with no words, only images, or that purport to appraise giftedness by allowing students to manipulate blocks or shapes). In other states, students falling just short of the cut-off may qualify if they have a high GPA - no matter the rigor of their course load - or show evidence of "leadership" or community service. It's one thing to raise the level of minority participation in these programs by strengthening the educational foundations of minority youngsters. But who really benefits if "gifted" programs ease their entrance standards or dilute their definition of academic excellence?
"Boosting minorities in gifted programs poses dilemmas," by Daniel Golden, Wall Street Journal, April 7, 2004 (subscription required)
Checker Finn's editorial on "The Discipline Paradox" discussed several difficulties of maintaining classroom order - particularly when a classroom contains some students who don't want to learn. Having served on the Fairfax County School Board for eight years, sitting on literally hundreds of expulsion panels (and reviewing the written record of hundreds more), I was able to see firsthand the discipline challenges that some teachers and school administrators face each day. Below are two observations from my tenure.
First, in most classrooms, the worst threat to order is the "serial disrupter" - the student who acts out in class on a regular basis. Several years ago, in an effort to curb the behavior of these serial disrupters, Fairfax County (VA) devised a behavior modification program, whereby disruptive students are sent to "time out rooms" for the remainder of the class period or (in elementary schools) lesson. In the best of these rooms, students found themselves sitting on a hard chair, facing the wall and working on their lessons in a cubicle-like setting. What we discovered was that
the students who remained in the classroom after the disrupter was excused suddenly discovered they could actually absorb some learning with their "colleague" out of the classroom,
the disruptive student found it a very unpleasant place to go, and frequently engaged in "behavior modification" so as not to find him/herself back there, and
the student who was disrupting class because she didn't understand the work in the classroom (Flash: kids who can't read act out in class!) could use this opportunity for some one-on-one help in learning the subject at hand.
We also found that teachers and administrators generally liked the program. So too did many of the students, including those students who acted out because they needed help.
Sadly, Fairfax County decided to reduce the scope of the program due to a spate of budget cuts, concerns about overrepresentation of minority students in time-out rooms, and the uneven application of the program. To me it would have been more logical to reform a program that had shown positive results, rather than cutting it in half, but that is yet another story of school board (and superintendent) politics.
Second, we shouldn't underestimate the impact of IDEA on discipline. In Fairfax County, we had any number of cases in which students who committed equal offenses (i.e., bringing a gun to school, committing a mob assault) in the same incident received unequal punishments because one of them was, say, dyslexic. This is true even when the offense has nothing to do with the disability, because the provisions of IDEA demand a dual system of discipline in the schools - one for "regular kids" and one for special ed students, the latter requiring a much higher level of documentation and paperwork at every stage of the process, at the end of which educational services must continue to be provided regardless of the infraction.
Given the growing number of students identified as special education (over-identification is another issue, to be sure), the paperwork burden of IDEA creates huge administrative complexities for educators. Many educators find it easier simply to shy away from, rather than confront, the expensive one-size-fits-all bureaucratic hurdles they're required to clear to discipline a special ed pupil. What's worse, all students - disabled or not - figure this out real quickly; the result is a continual stretching of limits that are almost nonexistent to begin with these days.
Christian N. Braunlich is vice president of the Thomas Jefferson Institute for Public Policy and a former member of the Fairfax County School Board.
If America's history teachers were broadly educated, deeply knowledgeable about the content that they're responsible for imparting to students, and free to draw their information, textbooks, and other instructional materials from whatever sources they judge best, all within a framework of sound academic standards and results-based accountability - under that dreamy scenario there'd be no reason for Sandra Stotsky to tackle the study that yielded Fordham's newest report, The Stealth Curriculum: Manipulating America's History Teachers.
The reality, however, is that many history teachers don't know much history. And the textbooks on which they depend are vast, themeless compendia of dull, dated, and denatured information. (See A Consumer's Guide to High School History Textbooks for more information.) Thus has arisen an immense cottage industry to supply teachers with pre-digested "supplemental materials" and "professional development" as part of an effort to prepare them better to teach about difficult issues.
As usual, this enterprise began with laudable intentions. Post 9/11, for example, how could we reasonably expect teachers who had never studied Islamic history to explain it to their pupils, especially if their textbooks lacked pertinent information? How could we expect them to handle complicated and emotionally charged subjects like the Holocaust and figure out what lessons to distill? To escort youngsters safely through the thicket of political correctness and ethnic politics that now surrounds such formerly benign holidays as Columbus Day and Thanksgiving?
So we try to compensate and backfill. Innumerable organizations and agencies, public and private, large and small, commercial and non-profit, create and deliver "supplemental" materials and "in-service education" or "professional development" for teachers. School systems and state education agencies. Publishers of every sort. Advocacy groups. Universities, research centers, and think-tanks. Itinerant teacher trainers. Cable networks and film producers. It's a long list, engaging many people and spending many millions. (Nobody knows how much.) Some is subsidized by tax dollars or philanthropy. Some is baldly commercial. Much comes out of school system budgets.
Yet we know staggeringly little about how good these materials and workshops are, or whether the information they present is balanced and accurate. We know even less about their efficacy and intellectual integrity. This turns out to be a vast dark continent within our education system.
It's also a troubled continent. Sandra Stotsky spotted the problem during her tenure as senior associate commissioner of the Massachusetts Department of Education (1999-2003) as well as during a distinguished career in education research, teacher preparation, and the development of academic standards. She began to collect examples of supplemental materials and professional development workshops aimed at K-12 history and social studies teachers. She attended some of those workshops and summer institutes. And she grew ever more alarmed by what she found.
It appeared that the creation of many of these "supplemental" materials, and the leadership of workshops by which teachers' knowledge is supposedly enhanced, had fallen into the eager hands of interest groups and ideologues yearning to use America's classrooms to shape the minds of tomorrow's citizens by manipulating what today's teachers are introducing into the lessons of today's children. Yet this was happening with little or no public awareness. In effect, the K-12 social studies curriculum was being subtly politicized by adult interests working outside the closely scrutinized domains of statewide standards, textbooks, pre-service teacher preparation, and state certification.
For this report, Stotsky separated the terrain into two parts, one dealing with supplemental materials, the other with professional development workshops. The shortcomings she spotted vary by topic, of course. But most share these features: under the guise of heightening teachers' awareness of previously marginalized groups, they manipulate teachers (and thus their pupils) to view the history of freedom as the history of oppression and to favor cultures that don't value individual rights over those that do.
Is there a remedy? Stotsky would wipe out much of this "supplemental" stuff and replace it with something very different. Alternatively, she suggests several shrewd ways of mitigating the problems if this enterprise persists.
So far, so good. We should certainly seek to compensate for weaknesses in the knowledge base of today's teachers while shielding them from manipulative mischief and reducing their risk of becoming unwitting pawns of ideologues. Over the long haul, however, we must insist that future teachers be better educated from the get-go or, as NCLB puts it, "highly qualified" in the subjects they will impart to children. Nowhere is this more important than in history.
But better-educated teachers ought not be equated with more time in ed school, maybe not even on campus (although well-conceived history courses taught by first-rate historians are hard to beat). People can also teach themselves history, pick it up from reading, the History Channel, even movies. The key is to insist that, however they learn it, tomorrow's teachers must know it - and prove it - before confronting children in the classroom. It may be sufficient to insist that they pass rigorous subject-matter tests, such as the American Board for Certification of Teacher Excellence. They can prepare for such exams however they like, in universities or elsewhere.
As with children, let's stop endlessly forgiving, compensating and remediating teachers. Let's do it right the first time. Until we do, the stealth curriculum may swamp the one we think our schools are teaching - and our teachers will remain vulnerable to manipulation by people and organizations who do not place America's best interests at the top of their priorities.
The Stealth Curriculum: Manipulating America's History Teachers, Sandra Stotsky, Thomas B. Fordham Foundation, April 2004
"Group blasts supplemental history teaching materials," Education Daily, April 14, 2003 (subscription required)
It's no secret that some states and districts have threatened to decline federal Title I funding to avoid the accountability provisions of NCLB. That is, of course, their choice to make. The state of Nebraska, however, has taken a more underhanded route, working hard since the passage of the law to have it both ways by doing just enough to keep its federal funds while skirting the spirit of the accountability provisions. Education commissioner Douglas Christensen has been instrumental in this effort by helping to convince the feds to approve what Tracy Dell'Angela of the Chicago Tribune calls "the nation's most unorthodox assessment system, which allows school districts to use portfolios to measure student progress." Nebraska's approved plan, the state's 517 districts will "design their own assessment systems: a portfolio of teachers' classroom assessments, district tests that measure how well children are meeting locally developed learning standards, a state writing test and at least one nationally standardized test included as a reality check." According to Dell'Angela, federal officials approved this system because Nebraska's state constitution "guarantees local control over school accountability and the state was able to demonstrate that the assessments were valid and reliable." Unfortunately, not only is such an assessment regimen expensive and time-consuming, it also consists predominantly of subjective measures of student achievement and makes comparisons among districts all but impossible - meaning that a student who passes the reading assessment in one district might not be able to meet standards in another.
"Nebraska dodges NCLB test requirements," Washington Times, April 12, 2004
"Nebraska shuns state tests," by Tracy Dell'Angela, Chicago Tribune, April 5, 2004
Between 1971 and 2000, the average GPA at Princeton rose from 2.99 to 3.66. By 2002, only 5 percent of seniors graduating from that eminent institution earned less than a B-minus cumulative GPA. Such grade inflation is not atypical, particularly among America's elite schools and universities, nor is recognition of the problem new. In fact, it was such a cause for concern back in 1998 that Princeton's Faculty Committee on Examinations and Standing called for the faculty to take "collective responsibility for halting grade inflation and grade compression at Princeton." Now, Dean of the College Nancy Malkiel proposes to "establish a common grading standard across the university of less than 35 percent A's for undergraduate courses and less than 55 percent A's for [junior and senior] independent work." The plan is designed to create a university-wide system to curb grade inflation "because no single department has any incentive to act unilaterally to address" it on their own. Not surprisingly, many students who benefit from the inflated grades are grumping. "If this happens, I'll never get an A again," one mused. Kudos to the honest student who said "I like the grade change because I know I've gotten A's on various tests and things that I shouldn't have gotten them on." (One hopes those A's weren't in English composition.)
"University takes steps to limit grade inflation," by Jeff Milgram, Princeton Packet, April 9, 2004
After years of trying to squelch charter schools and other choice efforts, the National Education Association seems to have decided: "If you can't beat 'em, join 'em." Beginning last October, in its publication California Educator, California's NEA affiliate - the California Teachers Association - began to promote a new effort to organize employees in hundreds of charter schools throughout the state. Articles featured such trademark NEA subtleties as "Without Union Protection, Teachers Are Vulnerable," and identified some of the primary objectives of CTA's organizing initiative: to secure "collective bargaining rights for all charter school teachers either by incorporating them into existing bargaining units or creating new bargaining units for them," to "deal with the privatization issues raised by charter schools," and to develop "programs for charter school teachers in the areas of advocacy training, communications, budget analysis, leadership, and professional development." Charter school advocates are understandably wary. Caprice Young, president of the California Charter Schools Association, criticized the union's effort to convene focus groups, complaining that "there have been cases where [the union has] paid teachers over $100 a day to take part in a focus group, and they're not really a focus group." Instead, Young alleged, those running the group were "basically trying to convince [charter school teachers] that they are unhappy." Young also blasted the CTA for "spreading misinformation about what charter schools are." And Dennis Snyder, a former public school teacher and 30-year union member who now teaches at Escondido Charter High School, blasted the union's efforts in California saying it's "about power, and getting more money into their coffers so they have more money to influence legislation. I think they should stay out of other people's houses."
"Calif. union to organize in charters," by Caroline Hendrie, Education Week, April 14, 2004
Andrew J. Rotherham, Progressive Policy InstituteMarch 2004
In this policy report, Andrew Rotherham, director of the 21st Century Schools Project at the Progressive Policy Institute, argues for incentives to lure National Board Certified Teachers (NBCTs) to teach in struggling schools and work with struggling students. NBCTs, he contends, "are disproportionately working in more affluent schools that are less likely to be struggling or having trouble meeting NCLB's teacher quality mandate." In response, Rotherham suggests a sharp increase in the bonus and pay differential for NBCTs and linking these incentives and rewards to state efforts to "help hard-to-staff schools meet NCLB's highly qualified teacher mandate or otherwise help struggling schools improve." Of course these recommendations are only as good as National Board Certification itself, and more research is needed to determine how well this certification identifies effective teachers, especially those who are most likely to be successful in needy schools. (See http://www.edexcellence.net/gadfly/issue.cfm?issue=19#201 for our review of the most recent study of NBPTS certification.) To see for yourself, go to http://www.ppionline.org/ppi_ci.cfm?knlgAreaID=110&subsecid=135&contentid=252498.
RAND Corporation2003
RAND-Education is the source of this Carnegie-funded 160-page monograph by Daniel F. McCaffrey, J.R. Lockwood, Daniel M. Koretz and Laura S. Hamilton. Given today's keen interest in value-added evaluations of teachers and schools, you'll want to know about it, even though it's boring and academic and concludes that more research is needed. Still, it does a nice job of setting forth possible sources of error in value-added teacher evaluations and analyzing well-known methods for doing this (including the celebrated methodology of Bill Sanders). In the end, sounding like a (juiceless) version of Churchill discussing democracy, the authors conclude that, notwithstanding the many perils of value-added analysis, it may be no worse than "other methods for using test scores to evaluate schools or teachers" and might even "provide less-biased and more-precise assessments of teacher effects." See for yourself at http://www.rand.org/publications/MG/MG158/.
Mass Insight Education
April 2004
Mass Insight Education has released a "bicoastal" poll that shows the public in two states wants more math in K-12 education - and has less "math phobia" than most people would believe. The poll, conducted by MIE and the Seattle-based Partnership for Learning, a business group, polled the public on attitudes toward math education in the Massachusetts and Washington State, and found that 75 percent believe that all students should take algebra and geometry, while one-third think all students should take trigonometry and calculus. The poll also looked at the attitudes of business leaders in both states, who say overwhelmingly that students' math skills aren't up to snuff, especially for work in high-tech industries. The public also likes standards-based accountability measures and thinks they've helped to raise student learning. The poll doesn't do much that the American Diploma Project (http://www.achieve.org/achieve.nsf/AmericanDiplomaProject?OpenForm) hasn't done in more detail. But it's a useful temperature-taking exercise and adds additional evidence to support that project's central contention: that a high school diploma is not the useful marker for skills and knowledge that it used to be. You can read the full report at www.massinsight.org.