We know that it’s hard to fire poorly performing teachers, especially after they earn tenure, so the more that districts can do to predict effectiveness ahead of time, the better.
This study by CALDER researchers Paul Bruno and Katharine Strunk examines whether a new teacher hiring and screening process in Los Angeles Unified School District (LAUSD), the second biggest district in the country, is actually ushering in more effective teachers.
Instituted in 2014–15, the Multiple Measures Teacher Selection Process is a standardized system of hiring with eight components whereby eligible candidates (those completing the application packet and meeting certification requirements) are scored on multiple rubrics. The eight components are a structured interview; professional references; sample lesson; writing sample; undergraduate grade point average; subject matter licensure scores; background (such as prior teaching or leadership experience); and preparation (such as attendance at a highly ranked college, evidence of prior teaching effectiveness or major in a credential subject field). The study uses a wealth of applicant data from 2014–15 through 2016–17, as well as teacher- and student-level administrative data for teachers who are ultimately hired and for their students.
Bruno and Strunk observe those individuals who pass the selection process, which typically means scoring a minimum of 80 percent, are considered eligible for hiring (totaling about 5,500 applications). The eligibility pool also includes about 10 percent of applicants who were granted an exception to the minimal passing requirement—both because principals can request that failing candidates be added back, and because failing applications are given a blind review by human resources staff and some are subsequently added back into the pool. Relative to the methodology, the analysts concede that the screening scores may serve as proxies for characteristics that administrators are actually observing in the interviews. Yet that’s not entirely bad if the process is also illuminating principals’ revealed hiring preferences through the interview rubric. Further, they conduct a number of statistical tests primarily intended to measure sorting bias—for example, new hires may accept easier placements—but find it is a minimal threat.
Results show that applicants who perform even better on the selection process (higher than 80 percent) are more likely to be subsequently employed as teachers, even though principals do not know their exact screening score. In addition, overall performance significantly predicts teacher outcomes once hired, including attendance, contributions to student achievement, and ratings on final performance evaluations. Specifically, a one standard deviation (SD) increase in overall score is associated both with teacher level value-added that is 16 percent of a SD higher in ELA and with as much as 73 percent lower odds of an unsatisfactory rating among all elementary teachers. Yet overall performance is not predictive of teacher retention. Moreover, the individual components of the selection process are differentially predictive of different teacher outcomes. For example, applicants are more likely to be hired if they have higher interview scores, sample lessons scores, or writing scores (among others) but undergraduate GPA and subject matter scores are not predictive. Finally, those who fail to meet the minimal score and are granted exemptions are less likely to be employed by LAUSD, despite the fact that they were actively chosen by one or more individuals for the exemption.
Clearly there’s a lot here to chew on but inquiring minds of district officials want to know: Can we hone these somewhat comprehensive hiring protocols to pinpoint what is most important in predicting effectiveness? Yet the study finds that when you tinker too much in an attempt to predict one particular outcome (like value-added, for instance), it greatly reduces the ability to predict other important outcomes such as attendance and retention. In other words, if you try to go too narrow on the selection process, it may come back to haunt you. Because, after all, forecasting teacher greatness ain’t easy.
SOURCE: Paul Bruno and Katharine O. Strunk, “Making the Cut: The Effectiveness of Teacher Screening and Hiring in the Los Angeles Unified School District,” CALDER (January 2018).
