The Education Gadfly Weekly

Corinne Herlihy, James Kemple, Howard Bloom, Pei Zhu, and Gordon Berlin
MDRC
June 2009

MDRC, one of the contractors that worked on the Institute of Education Sciences' final impact study of Reading First (RF), takes a second look at said study in these eight pages. In sum, RF may have been more successful than IES let on. First, MDRC explains, the timing of RF's grants made it harder for the program to show an impact. The first sites to get their RF money were also the sites that happened to be already using curricula in the mold of RF, which meant that the schools that had the longest window for RF to have an effect were already using RF materials. In fact, many districts were already using RF-recommended curricula by the time the program was fully implemented, thus limiting the effects that could specifically be attributed to RF. These schools, unsurprisingly, showed little improvement as a result of RF. The schools that were not already using RF-like curricula typically got their money later. These schools, which usually served disadvantaged students, had much greater room for improvement with RF's help. Unfortunately, IES's study methodology didn't account for the differences between these "early award sites" and "late award sites." Second, the actual 7-10 minute increase in "core elements of scientifically based reading instructional time" seen in most RF schools was too small to have a significant impact. (But, in schools where the time increase was substantially greater than average, the program did see gains in reading comprehension.) Overall, MDRC concludes that RF taught us three things: The recommendations of the National Reading Panel, which advocated the use of scientifically-based reading instruction, remain valid; it is possible to change teachers' instructional practices; and that improving student reading comprehension is difficult, and RF may have not been intensive enough. Despite the untimely and tragic death of RF, we should not be hopeless about the effects of quality reading programs, and perhaps some states and districts will see fit to spend part of their stimulus largesse on these effective initiatives. Read it here.

Center on Education Policy
June 2009

In this review of post-NCLB state test scores, CEP questions the fate of low- and high-performers under current "proficiency" metrics. Has an emphasis on a quantifiable middle point meant ignoring the tail ends of this bell curve? The answer, according to this study, is no; low-achievers and high-achievers aren't doing worse on average, even as NCLB encourages schools to zero in on the "bubble kids" just below proficiency. In particular, the study suggests that students in all age groups, in all subjects, and at all achievement levels are scoring at higher levels in 2008 than in 2002, at least on state tests. Those in the "proficient-and-above" category tended to see the largest gains, though that might be because state tests are designed to measure the performance of students near the "proficiency" line most accurately. However, state tests may not be the best way to examine gains by high-achieving students. Because most tests are set at laughably low levels, the best students tend to "top out" on them; we can't get an accurate read of their performance. Furthermore, the study merely reported whether there are more or fewer students at the "advanced" level (itself a target that's not necessarily all that advanced in most states), not whether the top students themselves made significant gains over time. This is a profound distinction, and one that is addressed in Tom Loveless's Fordham study from last year, which used NAEP data to track gains for the top ten percent of students; he found them to be "languid." Read the CEP report here.

Teachers from myriad Philadelphia high schools are complaining of palpable pressure to pass undeserving students on to the next grade. It comes in the form of memos, meetings, and even personal phone calls. "We have to give fake grades," revealed one teacher. "We're not asked to educate our kids. We're asked to pass them," explained another. "I'll get a phone call saying, ‘Are you sure he earned a 58? Are you sure it wasn't a 65?'" noted a third. A June 7 story in the Inquirer prompted Philly superintendent Arlene Ackerman to disavow the practice and order an investigation. One early casualty is a ludicrous district policy that set a minimum grade for failed assignments at 50, instead of zero. (Ackerman axed that amid the brouhaha.) Also blamed is the amount of paperwork required to fail a student, while passing holds no similar bureaucratic hurdles. Most troubling is that pass rates on courses, which teachers allege have been twisted into an assessment of their effectiveness, are a contributing metric to school report cards. The result, of course, is a classic perverse incentive for schools to move lagging students onward rather than reporting the truth about their performance. Though this situation seems a quagmire, the morale of this story is not: Social promotion does a grave disservice to students, to schools, and to teacher morale and professionalism.

"Teachers cite intense push to promote," by Kristen A. Graham and Martha Woodall, Philadelphia Inquirer, June 21, 2009

There's some consternation within the education establishment right now with what it sees as Arne Duncan's obsession with charter schools. There he is, warning states that they will lose out on "race to the top" funds if they don't eliminate their charter school caps. There he is, arm-twisting legislators in Tennessee to pass a stronger charter law. There he is, speaking at the National Charter Schools Conference about the key role that charters are expected to play in the stimulus-driven transformation of our system, including the reconstitution of failed district schools.

But to the perturbed establishment I say: Take the long view. The fact that the school reform world is so invested in the charter movement will help you over time, if only because said reformers are learning how hard it is to boost student achievement and how unjust first-generation accountability systems are for gauging school performance.

This isn't an entirely new development. Five years ago, when the New York Times published a front-page, AFT-seeded story about NAEP results that showed charters to be trailing their district-operated peers, charter advocates suddenly discovered "value-added assessment." It wasn't fair, they argued, to base judgments of school performance on such one-time "snapshots." Schools needed to be judged on the progress of their students over time. To which the establishment replied, "Hey, we've been saying that for years about No Child Left Behind. Welcome to the party."

And so it came as no big surprise when then Secretary of Education Margaret Spellings allowed a few, and eventually all, states to move to a value-added system of accountability under NCLB. With both establishment and insurgents agreeing on the inadequacy of one-point-in-time measures, moving beyond them became politically inevitable.

Fast forward five years, and consider how things have changed and how they have stayed the same. The insurgents have become the new establishment. Arne Duncan and most of his team are charter school enthusiasts, many of them having toiled in the fruitful vineyards of reform-minded outfits like the Gates Foundation and the NewSchools Venture Fund. They run the show now, at least in Washington. They have big bucks to throw around. And they are doing all they can to help their colleagues in the charter movement, particularly the corner of that movement that's about replicating schools with high test-scores and "bringing them to scale."

This week brought another critical charter study, one that's harder to debunk than the earlier Times NAEP analysis. (See more below.) In part, that's because it was done by CREDO, Macke Raymond's outfit at Stanford, and she's a known charter supporter. In part, it's because the study looks at progress over time, with a sophisticated (though by no means perfect) methodology. And its results are sobering: 37 percent of the charter schools in her sample trailed similar district schools in math gains over time. Only 17 percent of charters bested their district-run counterparts. (Yes, the news looks much better for charters if you ignore brand-new schools, or students who just switched schools. And yes, charters seemed to do especially well with poor kids.)

Still, many charter supporters will agree with Secretary Duncan that the CREDO study is a "wake-up call" and that "the charter movement is putting itself at risk by allowing too many second-rate and third-rate schools to exist." We will close down the bad schools and multiply the good schools, they shout. And we could. And should. And in some places, already do.

But will we shut down 37 percent of all of the charter schools in the country because they perform worse than their traditional counterparts on math state tests? Or 83 percent of all charters because they perform worse or no better than their district peers? Let's assume that the CREDO study was overly pessimistic. Will we kill twenty-five percent of charters? Fifty percent?

No. It's more likely that the charter movement will go in another direction, already foreshadowed by the reaction to the CREDO study. Advocates will argue, first, that charter schools are getting decent results with a lot less money than traditional public schools. Show us more money and we'll show you better results! Second, charters will point to broader measures of success than test scores alone. Look how engaged students are in these smaller settings, they will say. Consider the award-winning character-education program. See how effective we are in preparing students for college. Notice the family atmosphere. Consider our arts program, our focus on the "whole child," and on and on.

Such arguments aren't crazy. The "intangibles" of schools certainly matter as much to affluent families as test scores--maybe more. Why shouldn't they matter to poor families, and policymakers who share their concerns?

Which brings us to the paper on school accountability released today by the "Broader, Bolder Approach to Education" crowd. Are you ready for some unexpected news? It is eminently sensible.

That's a big surprise, for in the past this coalition has appeared eager to refight old battles about whether schools can be expected to help poor kids reach high standards. Now, however, it's arguing for a broader look at school success--what might be termed "test scores-plus." They would keep test-based accountability, tweaked in various ways (with progress-over-time measures, better assessments, a more robust NAEP, etc.) and supplement it with school inspectors. These inspectors would guard against lousy practices, such as "an undue emphasis on test preparation," and catch schools engaged in good ones, like "a collegial professional culture in which teachers and administrators use all available data in a collaborative fashion to continuously improve the work of the school."

It's not hard to image charter advocates supporting such a system and believing that it will show their schools to have more supportive learning environments than what is found in a typical public school. Studies like the one from CREDO might show lackluster test results, but look at the whole picture and the story is much brighter, they will say.

So public education groups: If you seek a kinder, gentler approach to school accountability, keep rooting for Arne and company to continue their magnificent obsession with charters. Because the more real-life, in-the-schools experience the reform crowd gains, the more appealing these second-generation accountability systems are likely to be to them. And with the "new establishment" already calling for "more money"--well, their agenda is your agenda, too.

A school in Orlando, Florida, is giving new meaning to the adage "visualize your success." For Mollie Rae Elementary, which went from a F to an A on the state rating system in just one year, focusing on pupil achievement was just one part of the winning formula. Turns out the school needed actual visual help, too, in the form of prescription eyeglasses for many of its pupils' pupils. "We tested 100 percent of our (530) students with the help of [University of South Florida]," explained principal Kathryn Shuler. "We found that some of the students weren't improving because they couldn't see." Squinting students couldn't make out the homework on the board or the words in their text books. If only school improvement were always this straightforward. Unfortunately, Mollie Rae's pupils might not be the only ones who need some ocular assistance. With 62 percent of Florida public schools receiving As on state metrics, but only 23 percent making Adequate Yearly Progress according to NCLB calculations, state administrators might want to check their own glasses.

"Glasses Help School Go From F To A," Click Orlando, June 18, 2009

While their large neighbor to the south shrinks the teacher pool, Connecticut legislators look to expand it, by creating more paths to get teachers into the classroom. In a special session last week, they passed a bill containing several laudable provisions: Teachers who've completed alternative certification programs and want to teach hard-to-staff subjects will be able to sit for a competency exam rather than being required to complete further class work; Teach For America will be able to expand beyond its current three school districts (Bridgeport, Hartford, and New Haven); and teachers certified in other states will have fewer hurdles to clear in order to teach in Connecticut. "We've just opened up major avenues to bring great teachers into the classrooms where they are needed," said lawmaker Andrew Fleishmann. The outcome didn't look as rosy at the end of regular session, when the bill stalled in the face of partisan wrangling. But further review by lawmakers--and a fervent advocacy effort led by school-reform powerhouse ConnCAN--propelled the bill to the governor's desk. This end-of-the-school-year victory promises a better start for Connecticut students come autumn.  

"Legislature Approves Proposed Teacher Certification Changes," by Daniela Altimari, Hartford Courant, June 21, 2009

"Key education bill passes in special session," by Elizabeth Benton, New Haven Register, June 22, 2009

Center on Education Policy
June 2009

In this review of post-NCLB state test scores, CEP questions the fate of low- and high-performers under current "proficiency" metrics. Has an emphasis on a quantifiable middle point meant ignoring the tail ends of this bell curve? The answer, according to this study, is no; low-achievers and high-achievers aren't doing worse on average, even as NCLB encourages schools to zero in on the "bubble kids" just below proficiency. In particular, the study suggests that students in all age groups, in all subjects, and at all achievement levels are scoring at higher levels in 2008 than in 2002, at least on state tests. Those in the "proficient-and-above" category tended to see the largest gains, though that might be because state tests are designed to measure the performance of students near the "proficiency" line most accurately. However, state tests may not be the best way to examine gains by high-achieving students. Because most tests are set at laughably low levels, the best students tend to "top out" on them; we can't get an accurate read of their performance. Furthermore, the study merely reported whether there are more or fewer students at the "advanced" level (itself a target that's not necessarily all that advanced in most states), not whether the top students themselves made significant gains over time. This is a profound distinction, and one that is addressed in Tom Loveless's Fordham study from last year, which used NAEP data to track gains for the top ten percent of students; he found them to be "languid." Read the CEP report here.

Corinne Herlihy, James Kemple, Howard Bloom, Pei Zhu, and Gordon Berlin
MDRC
June 2009

MDRC, one of the contractors that worked on the Institute of Education Sciences' final impact study of Reading First (RF), takes a second look at said study in these eight pages. In sum, RF may have been more successful than IES let on. First, MDRC explains, the timing of RF's grants made it harder for the program to show an impact. The first sites to get their RF money were also the sites that happened to be already using curricula in the mold of RF, which meant that the schools that had the longest window for RF to have an effect were already using RF materials. In fact, many districts were already using RF-recommended curricula by the time the program was fully implemented, thus limiting the effects that could specifically be attributed to RF. These schools, unsurprisingly, showed little improvement as a result of RF. The schools that were not already using RF-like curricula typically got their money later. These schools, which usually served disadvantaged students, had much greater room for improvement with RF's help. Unfortunately, IES's study methodology didn't account for the differences between these "early award sites" and "late award sites." Second, the actual 7-10 minute increase in "core elements of scientifically based reading instructional time" seen in most RF schools was too small to have a significant impact. (But, in schools where the time increase was substantially greater than average, the program did see gains in reading comprehension.) Overall, MDRC concludes that RF taught us three things: The recommendations of the National Reading Panel, which advocated the use of scientifically-based reading instruction, remain valid; it is possible to change teachers' instructional practices; and that improving student reading comprehension is difficult, and RF may have not been intensive enough. Despite the untimely and tragic death of RF, we should not be hopeless about the effects of quality reading programs, and perhaps some states and districts will see fit to spend part of their stimulus largesse on these effective initiatives. Read it here.