The Education Gadfly Weekly

Patrick Wolf, Babette Gutmann, Michael Puma, Brian Kisida, Lou Rizzo, Nada Eissa
Institute of Education Sciences
March 2009

Amid all the sound and fury surrounding the D.C. voucher program, this study is a significant feather in the cap of the program's supporters. Why? Because despite the study's rigorous methods (a gold-standard randomized controlled trial, which usually finds "no effect"), students offered a voucher were performing at statistically higher levels in reading after three years (equivalent to a 3.1 month gain) than students not offered a scholarship. The reading finding is even more striking since the treatment group was highly mobile--a factor that likely contributed to the null findings in years one and two. (Over the three years, in fact, 51 percent of the treatment group switched schools 2-3 times.) Unsurprisingly, both groups performed similarly in math and the program did not have a significant impact in reading or math for those students who applied from the worst performing public schools. While this latter finding is unfortunate and has been cited by Secretary Duncan as reason to shut down the program, we should remember that students coming from the very worst schools are the hardest to remediate and require the most time to do so; this study only measures three years. In addition to the headline finding, the study discovered that the voucher program had improved reading achievement for 5 of 10 (statistically significant) subgroups: students not previously attending a school in need of improvement, those with higher levels of performance at time of entry into the program, those entering grades K-8 when they applied (i.e., everyone but high school students), female students, and first year students (though these last two are less reliable than the other three). Shame on the Department for pulling its Friday-afternoon-release stunt for such important data. We're hoping Congress perused the Saturday papers. Read it here.

Kristine Lamm West and Elton Mykerezi
University of Minnesota, Twin Cities
Presented at National Council on Teacher Quality conference, "Help or Hindrance? The Impact of Teacher Roles, Rules and Rights on Teacher Quality," March 26, 2009

This study, which won first prize at the National Council on Teacher Quality's recent conference, used NCTQ's Teacher Roles, Rules and Rights database (TR3) and 1999-2000 School and Staffing Survey (SASS) data to evaluate the impact of unions on various elements of teacher compensation. The two complement each other well; TR3 is newer (the data is from 2006-2007) and more specific (including indicators such as salary schedules and teacher performance evaluation methods) but only includes America's 100 largest districts. SASS is much larger and designed to be representative on the state and national level, but is older and less specific. The authors' findings are intuitive, but interesting nonetheless. For example, the frequency of any type of performance pay scheme is the same for unionized (29 percent) and non-unionized (28 percent) districts. The incidence of output based performance schemes (those with at least one indicator based on student outcomes), however, is significantly lower in unionized (16 percent) than non-unionized (25 percent) districts. In fact, unionization of a district decreases the probability of a pay plan that rewards student outcomes by 81 percentage points. That number increases to 84 percentage points when the pay plan is based exclusively on student outcomes. The authors also discover that benefits (defined by SASS as district contributions to Social Security and other taxes, medical, dental, unemployment etc.) typically make up 25 percent of total salary and that states with strong cultures of unionization amongst all professions are more likely to have teachers' unions. All of this and more can be found here.

In case it wasn't clear that teachers' pensions are about as sustainable as daily print newspapers, New Jersey is here to remind us. The Garden State Teachers' Pension and Annuity Fund, which covers retirement benefits for about 232,000 active educators, retirees, and beneficiaries, is facing a whopping $14.1 billion long-term shortfall. And that was as of June 30, 2008. In the nine months since, the state's retirement investments lost 25 percent of their value. We've long argued (see here and here, too) that the current teacher pension system is both untenable and perversely incentivizes teachers to stick it out after they've checked out (or, conversely, to retire while they still have some fire). There are two main ways to fix this problem: take even more money out of the pockets of today's teachers and taxpayers in order to shore up the system, or bring retirement benefits back down to earth. At a time when private sector employees are seeing their 401(k)s go up in smoke, we think the first strategy will prove as politically popular as the AIG bailout. Isn't it time for Plan B?

"NJ teachers pension fund closer to insolvency," by Jonathan Tamari, Philadelphia Inquirer, April 3, 2009

Do you ever dream about what you'd do if you were Secretary of Education? If you're a teacher, no doubt you'd work to make federal policy more teacher-friendly. If you're a researcher, you'd strive to make it more evidence-based (and to increase the R & D budget). And if you're a big-city superintendent? Of course, you'd do exactly what Arne Duncan is doing: push for maximum flexibility at the district level while attacking federal and state policies that impede your reforms.

Consider the litany of superintendent-friendly policies that Duncan pursued last week, through his stimulus guidance, his intentions to adjust the Bush Administration's Title I regulations, and his comments to the press.

First, he'll now allow districts "in need of improvement" to offer "supplemental services" (free tutoring) directly, rather than outsourcing all of it to private providers. (His predecessor had already granted this permission to Chicago and a few other cities.) Second, he won't give districts a hard time if they can't provide timely notice to parents that their children are eligible to switch schools under No Child Left Behind, at least not if it's the states' fault (for not releasing timely test score data). Third, he called attention to a feature of the American Recovery and Reinvestment Act that will enable districts to spend the majority of their stimulus dollars almost any way they like, including school construction. And fourth, he talked up the benefits of Chicago-style mayoral control of big-city schools, as a strategy to consolidate power and authority in the hands of one district leader.

This all makes a certain amount of sense. Duncan was right when he told the Washington Post that "we need to have a tight, clear bar that we are all shooting for...but provide much more flexibility and the ability to innovate and be looser in how folks get there."

But what's perhaps most significant is the Superintendent-Secretary's approach to states. For decades, governors have been seen as major drivers of education reform. But now, rather than expecting states to lead, Duncan is asking them to get out of the way. This can be glimpsed in two important developments. First, the Secretary made it clear that states will have almost no discretion over the spending of stimulus dollars, which will go out via formula to the districts. And second, he put governors on notice that, if they want some of his $5 billion "race to the top" fund, they are going to have to strip barriers to local reform out of state law, such as charter school caps or rules disallowing districts from using student achievement data to inform decisions around teacher tenure.

Call it the Superintendents' Revenge. It must be satisfying to those at the helm of the country's school districts. And it sure makes sense if you're leading a system in Richard Daley's Chicago and Rod Blagojevich's Illinois. But whether it will work for the rest of the country, which isn't similarly blessed (or cursed), remains to be seen. Here's hoping for a dream come true.

The No Child Left Behind (NCLB) Act establishes the lofty goal of holding schools to account for all children achieving "proficiency" in reading and math by 2014--now barely five years away. As the nation now faces up to what to do next, it needs to squarely face the fact that NCLB's authors conjured four illusions--possibly because they were engaged in self-delusion.

Two of these have been well documented by the Thomas B. Fordham Institute and the Northwest Evaluation Association. First, the "high standards illusion," whereby states create the impression that most of their students are meeting expectations, but only by defining "proficiency" downward. And, second, the "equal treatment illusion," whereby NCLB gives the impression that we have a national approach to accountability; in reality, schools are treated quite differently in the various states. One that's deemed to be "failing" in state A can be seen as doing fine in state B. Thus, good schools may be branded failures and ineffective schools let off the hook, depending on where they're situated. The reputations of teachers can be unjustifiably harmed, and they can possibly lose their jobs. Students who need interventions may not get them and, since their schools may wrongly be identified as succeeding, could find themselves barred from transferring--as the law permits--from an ineffective school to a more effective one.

But two more illusions are also worth noting. First, there's the "identification of ineffective schools" illusion. NCLB creates the impression that it identifies "failing" (i.e. in need of improvement) schools. That does not happen, regardless of whether the standard is set high or low. The proficiency levels that schools are working to have students reach are based on point-in-time tests, given, for example, at the end of the eighth grade in reading. The results of that type of test reflect all student knowledge acquired since birth--from being read to, or not, in the home; from going to quality child care or not; from learning--or not--during grades K-7 and perhaps at a different school; and from having widely varying summer experiences in which reading achievement is gained or lost.

By the standard of Evaluation 101, this is no measure of the effectiveness or quality of a school, because it does not measure what the student learned during the eighth grade at that particular school. Also, since different students are involved from year to year, scores may go up or down because of population changes in the neighborhood--flight to the suburbs, for example.

This means that the consequences in NCLB are applied on nearly a random basis. Research has made clear that schools declared as failing on such point-in-time measures are a quite different set of schools than those declared to be failing on a "gain score" measure of the progress students made while in school.

Finally, there's the "prescribed remedies will improve education" illusion. The law creates the impression that it will trigger federally-required actions that make schools better and will raise achievement. For instance, when schools have not met their progress targets, students can transfer to schools that have. This remedy does not derive from research results or successful replication. If lower-scoring students transfer, do the scores in the schools they left go higher and students there show progress? And are the scores in the receiving schools dragged down by the lower-scoring students who transferred in, so that the "better" schools now become "worse" and get sanctioned the next time around? Of if higher-scoring students transfer, do scores in the schools they left sink lower even as they try to right the ship? At the time this was decided, no one knew how this might play out, and I suspect that no one knows today how it is playing out.

If a school fails to make the required progress over five years, the district must "restructure" it. Several options are specified, including the adoption of a remedy of the district's choice. This amounts to saying, "Don't just stay open; do something." These remedies are not tested approaches to turning schools around and are not based on knowledge of whether or not such moves improve schools and raise student achievement.

All of these realities need to be squarely faced; illusions will not suffice. Only when the right schools are identified for corrective actions and the right actions are taken will schools improve and student achievement rise. As the nation sets forth on a new education policy, there is a need to start from a full understanding of the current situation.

Paul E. Barton is an education writer and consultant. He is a former Director of the Education Testing Service (ETS) Policy Information Center and author of 'Failing' and ‘Succeeding' Schools: How Can We Tell?

While the phenomenon of unionized charter schools is only budding, parochial schools have a longer tradition of collective bargaining. But two Catholic schools in Staten Island may have found a way to shed the union albatross. They are part of a larger group of ten schools operated by the Archdiocese of New York that, we learned in January, are to become independent come fall. As independent schools, they'll have their own governing boards and, most importantly, be responsible for their own financial stability and self-sufficiency. (This seems to have been a successful tactic for Catholic schools in Boston.) The Lay Faculty Association, which represents about 400 teachers at the ten Archdiocesan schools, cried foul; by cutting the schools loose, LFA argued, the Church was really trying to undermine the union by diluting its contractual power. Now, the two soon-to-be-independent schools in Staten Island have gone a step further: all of their teachers will have to reapply for their positions once the transition takes place. While this doesn't do away with the union all together, it does sidestep the burdensome dismissal process, and demonstrate a private-sector business savvy long needed in these parochial institutions. Let's hope that merit, not tenure, is rehiring criterion numero uno. 

"Teachers at Farrell, Moore back to square one," by Amisha Padnani, Staten Island Advance, April 8, 2009

While D.C. Chancellor Michelle Rhee's battle over salary and tenure has gained much media attention, less heed has been paid to her plans to overhaul the District's teacher evaluation systems. Yet a consensus is growing that they need dramatic reform, too. Studies by the New Teacher Project in various cities found that a disproportionate percentage of teachers (in some cases, almost 100 percent) are given positive evaluations; Education Sector found that teacher evaluations are more like "drive-bys" than anything else--woefully short, vague, and based on indicators that employ criteria unrelated to student learning. Luckily, Rhee has retained a top notch research team (Harvard ed school researcher Tom Kane and the research organization Mathematica) to help DCPS develop a value added system; she's also invited DC's teachers to a series of focus groups on the subject and retained former National Teacher of the Year Jason Kamras to head this effort for her. While we don't know what Rhee's squad will come up with, we do know that the current evaluation system is a joke. Even the union can agree on that. But we have to start somewhere and Rhee's mettle to at least try to fix this mess--and recruit some smart people to help her--is commendable. 

"Rhee Works on Overhaul of Teacher Evaluations," by Bill Turque, Washington Post, April 7, 2009

To the long, familiar list of reasons urban education has failed--too-powerful teacher unions, underfunded and mismanaged schools, poverty's ill effects--Education Secretary Arne Duncan has added another: lack of "leadership from the top." He's talking about mayors in particular, and he wants more of them in charge of urban school districts, à la Chicago, New York, Boston, and Washington, D.C. How strongly does he feel about it? "At the end of my tenure, if only seven mayors are in control, I think I will have failed." Bold words, indeed. And he's willing to go the extra mile to press his case. He told the nation's mayors, "I'll come to your cities. I'll meet with your editorial boards. I'll talk with your business communities. I will be there." But as he surely knows, this reform, like all others, is no panacea. Mayoral control is only as good as its mayor.

"School chief: Mayors need control of urban schools," by Libby Quaid, Associated Press, March 31, 2009

Got sagging pants? Not if you go to Plantation High School in Broward County, Florida. That's because two teachers, inspired by President Barack Obama's comment last year that "brothers should pull up their pants," have launched a crusade against baggy offenders. The school recently held a "Pull up your Pants" day; a local Wal-Mart donated 200 belts to help in the effort. "If your pants are saggin' and you want to adopt a jailhouse mentality, that will show in your attitude toward everything," explained Diana Carter, a ninth-grade teacher who helped organize the event. "We're not trying to take their individuality away, but there's a time and a place for everything." The trend apparently has its roots in some unseemly places, not the least of which is jail, where prisoners' belts are confiscated so they won't be used as weapons or to commit suicide. Some celebrities also set a bad example when it comes to undergear exposure. For the most part, Plantation High's students acquiesced. One even claimed he'd given up the baggy look for good. Why? "I asked a girl if she liked it and she said not no more [sic]." Let's hope Carter's next crusade is pulling up some sagging grammar.

"Obama Inspires Florida Students to Pull Up Their Pants," Gil Kaufman, MTV.com, March 27, 2009

"Plantation High students get hand pulling up their pants," Hannah Sampson, Miami Herald, March 27, 2009

Kristine Lamm West and Elton Mykerezi
University of Minnesota, Twin Cities
Presented at National Council on Teacher Quality conference, "Help or Hindrance? The Impact of Teacher Roles, Rules and Rights on Teacher Quality," March 26, 2009

This study, which won first prize at the National Council on Teacher Quality's recent conference, used NCTQ's Teacher Roles, Rules and Rights database (TR3) and 1999-2000 School and Staffing Survey (SASS) data to evaluate the impact of unions on various elements of teacher compensation. The two complement each other well; TR3 is newer (the data is from 2006-2007) and more specific (including indicators such as salary schedules and teacher performance evaluation methods) but only includes America's 100 largest districts. SASS is much larger and designed to be representative on the state and national level, but is older and less specific. The authors' findings are intuitive, but interesting nonetheless. For example, the frequency of any type of performance pay scheme is the same for unionized (29 percent) and non-unionized (28 percent) districts. The incidence of output based performance schemes (those with at least one indicator based on student outcomes), however, is significantly lower in unionized (16 percent) than non-unionized (25 percent) districts. In fact, unionization of a district decreases the probability of a pay plan that rewards student outcomes by 81 percentage points. That number increases to 84 percentage points when the pay plan is based exclusively on student outcomes. The authors also discover that benefits (defined by SASS as district contributions to Social Security and other taxes, medical, dental, unemployment etc.) typically make up 25 percent of total salary and that states with strong cultures of unionization amongst all professions are more likely to have teachers' unions. All of this and more can be found here.

Patrick Wolf, Babette Gutmann, Michael Puma, Brian Kisida, Lou Rizzo, Nada Eissa
Institute of Education Sciences
March 2009

Amid all the sound and fury surrounding the D.C. voucher program, this study is a significant feather in the cap of the program's supporters. Why? Because despite the study's rigorous methods (a gold-standard randomized controlled trial, which usually finds "no effect"), students offered a voucher were performing at statistically higher levels in reading after three years (equivalent to a 3.1 month gain) than students not offered a scholarship. The reading finding is even more striking since the treatment group was highly mobile--a factor that likely contributed to the null findings in years one and two. (Over the three years, in fact, 51 percent of the treatment group switched schools 2-3 times.) Unsurprisingly, both groups performed similarly in math and the program did not have a significant impact in reading or math for those students who applied from the worst performing public schools. While this latter finding is unfortunate and has been cited by Secretary Duncan as reason to shut down the program, we should remember that students coming from the very worst schools are the hardest to remediate and require the most time to do so; this study only measures three years. In addition to the headline finding, the study discovered that the voucher program had improved reading achievement for 5 of 10 (statistically significant) subgroups: students not previously attending a school in need of improvement, those with higher levels of performance at time of entry into the program, those entering grades K-8 when they applied (i.e., everyone but high school students), female students, and first year students (though these last two are less reliable than the other three). Shame on the Department for pulling its Friday-afternoon-release stunt for such important data. We're hoping Congress perused the Saturday papers. Read it here.