Reid Lyon, former Reading Czar and one of the creators of Reading First, posted a comment about Shep Barbash's Education Next article that's so crucial to the current debate that it's worth excerpting at length:
The recent Reading First Impact Study interim report did some thing s correctly (employed a strong design for the questions they asked), but appeared to miss some very important confounds, leading me to have difficulties interpreting the results. First, the evaluation did not address all of the evaluation targets established in the law, thus narrowing the scope and comprehensiveness of the evaluation Congress intended. Second, and most importantly, the sample of states selected for inclusion in the study was not sufficient to test a number of variables that are critical to interpreting the data. As hard as I try, I cannot see how the sample would be considered representative. Third, the evaluation examined the effect of resources (Reading First funding) on a single measure of reading comprehension. As Steve Raudenbush has argued convincingly, an evaluation study comparing a group that received the resources versus another group that did not answers very little about the programs actual effectiveness or the ability of a study to inform improvements in the program or guide policy.There are many factors at the implementation and instructional level that have to be examined and studied to refine any interpretation of the main effect of no significant difference. As Mr. Stearns probably knows, many school districts that implemented Reading First in some schools implemented the same programs in non-Reading First schools. Professional development activities funded through Reading First were available to all schools in a district, not just Reading First schools. Most of the states that Mr. Barbash highlighted in his article where not included in the sample drawn for the impact study. Approximately 60 percent of Reading First and non-Reading First schools were implementing the same programs by the third year of implementation according to Tim Shanahan. There was a significant degree of contamination from the "bleeding" of programs and resources across Reading First and non-Reading First schools. Believe me, if the final impact study report clarifies all of these issues, that would be very helpful.
According to our own Checker Finn, just yesterday the Institute for Educational Sciences director Russ Whitehurst admitted in front of 200 people that the "bleeding" of Reading First methods to schools in the control group is a probable explanation for the findings of his evaluation. In other words, non Reading First schools were using Reading First's (proven and effective) methods too. That would have been a nice point to get across to the press before they wrote stories declaring the initiative a failure.
Let it be clear: even the government's chief education researcher is saying that the findings of his evaluation don't mean that Reading First is "ineffective" and should be scrapped. Congress, are you listening?
