A new, somewhat unsettling NBER working paper by Thomas Dee and colleagues examines the prevalence and implications of teachers tampering with student test scores on New York State Regents exams.
The analysts focus on exams taken between the 2003–04 and 2009–10 school years in New York City, which can be reliably linked to students. To qualify for a “local” diploma, the lowest degree available in New York, students entering high school before fall 2005 had to score at least a 55 on all five core Regents exams (English, Math, Science, U.S. History/Government, and Global History/Geography). In fall 2008, local diplomas were eliminated, and students were required to receive at least a 65 score on all five tests.
Up until 2012–13, Regents exams were graded by teachers from students’ own schools, and a policy was in place that required exams with scores just below the cutoff to be re-scored by the schools. The analysts document clear spikes around the cutoffs in an otherwise smooth test score distribution. In other words the scores immediately below the cutoffs appear less frequently than expected from a “well-behaved empirical distribution,” and the scores at or just above the cutoffs appear more frequently than expected, suggesting that scores just below the cut were bumped up. As a point of comparison, they show the spread for standardized math and ELA scores taken by students in grades 3–8, which reveal a smooth distribution; those exams are scored centrally by the state.
In analyzing the magnitude of the “manipulation,” they estimate that teachers inflated roughly 40 percent of test scores near the cutoff. Inflating scores for students who would have failed the test by a small margin raised the probability of their graduating from high school by approximately 27 percent. They also estimate that the black/white achievement gap would be about 5 percent larger in the absence of test score manipulation. Black and Hispanic students, along with those with lower baseline scores and worse behavioral records, benefited more from test manipulation in the aggregate due to the greater number of these students near the cutoff. Interestingly, they also find evidence of manipulation for higher cutoffs on elective exams, which provide students with benefits like taking advanced coursework, conferring college credit, and granting automatic admission to some public colleges. The analysts surmise that teachers are acting out of “altruism” for students, especially because there was no evidence that teachers manipulated scores to receive bonuses and no evidence that manipulation was impacted by test-based accountability.
In 2011, the New York State Board of Regents ended the practice of teachers scoring the exams of students at their own schools (and of rescoring in general) and moved to a centralized grading policy. Not long after, all of the tampering appeared to end. Analysts say that “manipulating scores may have been a ‘cultural norm’ among New York high schools, in which students were often spared any sanctions involved with failing exams, including retaking the test or being ineligible for a more advanced high school diploma.” Yet these are just the type of “norms” we should avoid teaching children. Life after high school, unlike these high school teachers, won’t be so kind.
SOURCE: Thomas S. Dee, Will Dobbie, Brian A. Jacob, and Jonah Rockoff, “The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations,” National Bureau of Economic Research (April 2016).
