The first set of preliminary findings from the Gates-funded Measures of Effective Teaching (MET) project generated much conversation—and some criticism. This latest report, also preliminary, is not much different. (Remember that this $45 million project seeks to ferret out, or design, an optimal teacher-evaluation system through the analysis of student test scores, surveys, and thousands of hours of classroom observations.) While the first iteration compared student scores with survey responses, this one analyzes the predictive strength of five frameworks for classroom observations (think D.C.’s IMPACT program for an idea of what they look like). The study finds that, while each method is positively correlated to pupil achievement (on both state tests and independent tests), the reliability of observations pales in comparison to value-added measures (VAM): The reliability of VAM is about double that of a single observation—from any of the tested measurement systems. Predictive abilities increase significantly when VAM and student-survey data are combined with classroom observations—leading the authors to recommend use of multiple measures when evaluating teachers. In response, Jay Greene has again sounded the battle cry. And perhaps rightly so. While the report’s analyses clearly show that value-added data is the single strongest predictive factor for student achievement and that adding observations almost negligibly improves reliability, nowhere do the authors caution policymakers about the potentially high cost and low yield that come with that addition. In other words, it is cheaper and almost as reliable to rely entirely on value-added data. (Gates does remind that classroom observations can help buttress strong teacher-improvement programs, though—in a way that VAM can’t.)This report deserves a close look. And expect the debate to be rekindled later this year when the study’s final report is due to be released.
Gathering Feedback for Teaching: Combining High-Quality Observations with Student Surveys and Achievement Gains (Seattle, WA: Bill and Melinda Gates Foundation, January 2012).
