"What? Me Worry?" Alfred E. Newman, Mad Magazine's mascot since the late 1950s, delivered this signature line whenever the world around him was going, well, mad. So, too, it seems, those working in the field of educational research.
That's the upshot of an important study by Peggy Hsieh and Joel R. Levin, which ran in the Journal of Educational Psychology and chronicles ed researchers' continued retreat from accepted research methodology. In this case, randomized experiments.
Randomized experiments, aka field trials, whereby an experimental group that receives an intervention (say, Whole Language) is compared with a control group that receives no intervention, have been standard operating procedure since rats were first run through mazes. But who needs control groups in the age of feelings-based research? Never mind that it's the theme song of Russ Whitehurst and the federal Institute of Education Sciences.
Hsieh and Levin report that "The percentage of total articles in these four journals [Cognition & Instruction, Contemporary Educational Psychology, Journal of Educational Psychology, Journal of Experimental Education, American Educational Research Journal] based on randomized experiments decreased over the 21-year period in both the educational psychology journals (from 40 percent in 1983 to 34 percent in 1995 to 26 percent in 2004) and the American Educational Research Journal (from 33 percent to 17 percent to 4 percent)."
To be sure, education is not the only field to succumb to the allure of fact-free debate. Jeffrey Pfeffer and Robert Sutton report in the January 2006 Harvard Business Review that "thousands of studies are conducted on medical practices and products every year. Unfortunately, physicians don't use much of it. Recent studies show that only about 15 percent of their decisions are evidence based." Pfeffer and Sutton add, "The same behavior holds true for managers looking to cure their organizational ills. Indeed, we would argue, managers are actually much more ignorant than doctors about which prescriptions are reliable-and they're less eager to find out."
There is, of course, abundant research available for those genuinely interested in effective practices. But the prescriptions arising from that research-eat less, exercise more, learn phonics, know your math facts, devote more time to literacy instruction, have people who know science teach science, etc.-are so much less appealing than the Ouija board hints that emanate from ersatz research from a variety of fields. Perhaps this explains why the doctors of medicine and business who write about education are frequently no more helpful than the doctors of education.
Why are randomized experiments being dropped faster than a tainted control group? Hsieh put that question to a number of folks. One "speculated that with the increasing popularity of qualitative methods (i.e., not relying on quantitative data), some researchers may have rejected the underlying assumptions of experimental research in favor of a post-modern, relativist view." A more cynical interpretation holds that because empirical research is difficult to conduct and yields unpopular results, many authors simply take their studies down an easier path. Why risk tenure by studying the effectiveness of phonics, for example, if a university promotion committee member worships at the altar of whole language? Why bother with multivariate analysis when a feminist critique of patriarchal statistical methods will do?
Universities and their faculty members are not the only culprits here. A bevy of publishers and other vendors benefit financially from research that is short-term, isolated, unscientific and devoid of context. Thus we often find assertions that "research proves" this or that program is associated with increased student achievement, even though other analyses show that the same program is associated with no such gains. Education policy makers and school system consumers are more likely to be in hot pursuit of the latest magic bullet than they are to make fundamental changes in course content, assessment rigor, daily schedules, and professional requirements for administrators and teachers.
In fact, no educational program exists in isolation. Thus, careless and isolated studies of those programs will yield misleading results. Does Success For All or Open Court or Read 180 or, for that matter, McGuffey's Reader really work? Typical research that considers only the presence or absence of these programs is unlikely to provide answers. The programs are effective when partnered with effective teaching, accurate feedback, and meaningful leadership support. When these contextual variables are absent, the programs usually accomplish little.
Unfortunately, it is the rare research study that provides long-term observation of student performance, teaching practices, and leadership support, and also provides a systematic measurement of other contextual variables. McGuffey's Reader will be a terrific program when implemented with the necessary time, diligence, and supervision. But the most elegantly conceived 21st Century reading program will fail without these essential elements. Consider one of the most basic contextual variables: time. One superintendent swore to me that he had effected "system-wide" implementation of a "proven" reading program. Yet a cursory review of the schedules of his 67 elementary schools revealed that teachers devoted from 45 to 180 minutes each day to the program. What really caused that district's success? The program or the time and attention provided by teachers and school leaders?
Karen Harris of Vanderbilt University, editor of the Journal of Educational Psychology, deserves credit, along with Hsieh and colleagues, for publishing an important study. They prove what many have long suspected: that the quality of research in education has deteriorated. This does not mean sound empirical research is unavailable. It just means that we have to dig deep, and plow through a lot of fluff in order to find the analysis we should expect.
Douglas Reeves is CEO and founder of the Center for Performance Assessment.
