Ohio has been a national leader in using value-added measures of student academic growth. The current value-added system was piloted in 2007 and fully integrated into the state accountability system in 2008. Yet, since then Ohio’s value-added system has come under increasing scrutiny and criticism from some district superintendents and others in the field.
Critics of Ohio’s value-added system have raised concerns about its methodology and its usability, while others have criticized the Ohio Department of Education for giving too much weight to value-added when it rates local schools and districts. We’ve been tracking these issues and reporting on them since August 2008, when we published Ohio Value-Added Primer: A User’s Guide. As Ohio moves toward launching teacher evaluation systems in the fall of 2013 that, by law, must incorporate value-added analysis where available, Ohio’s current value-added metrics warrant additional attention.
Ohio’s measurement of value-added is based on the SAS Institute’s Education Value-Added Assessment System (EVAAS). EVAAS develops a customized prediction of each student’s progress based, if possible, on the student’s own academic record as well as that of other students over multiple years, with statewide test performance serving as an anchor. In short, EVAAS measures student academic growth in reading and math for grades four through eight using a complex calculation. The state uses these measurements to assign schools and districts one of three ratings:
1. Above expected growth – indicates that the students in a school or a district made greater progress than expected. These schools and districts are “adding value.”
2. Met expected growth – indicates that students made the amount of expected academic progress in one school year. Districts and schools in this category are still adding value, but not as much as those schools rated Above expected growth.
3. Below expected growth – indicates that students in the school or district made less academic progress than the state expected.
EVAAS has been criticized for using a “black box” methodology that is proprietary and not transparent or readily understandable to outsiders. The SAS methodology is available to those who ask for it and it is indeed incredibly complex. Trying to understand it demands expertise that only a few of the country’s top psychometricians possess (see this report outlining SAS EVAAS Statistical Models). For teachers, policy makers, and the public, it certainly isn’t easily digestible or intuitive.
Another criticism of Ohio’s model comes from Professor Douglas Clay, of Cleveland State University, who wrote an editorial last year that highlighted the “yo-yo effect” anomaly. The yo-yo effect basically was a trend wherein a significant number of schools produced below expected gains one year, and then dramatically higher gains the next (or the reverse). Clay highlighted state data that showed more than 80 percent of Ohio schools achieving “below expected growth” in fifth-grade reading in 2008, but this yo-yoed to 98 percent of schools making “above expected growth” in sixth-grade reading one year later.
In response to Clay’s findings, ODE changed the value-added calculation to be more precise and to provide a better balance between the three achievement ratings. Specifically, the state now requires two standard errors of measurement, instead of one, to place schools in either the below expected or above expected growth designation. This pushes more schools into the middle category of “meeting” expected gains, making it harder to fall to either extreme. As a result, while the 2008-09 statewide value-added results looked more like a ski slope, with most schools meeting expected gains and far fewer falling below expected gains, the 2010-11 numbers more closely resembled a bell-curve with distribution of results packed mostly in the middle of the growth designations and then spread out equally on the two sides.
This modification results in the value-added metric providing a more accurate and balanced representation of student growth across schools and districts, but the process of tweaking the system to produce a more balanced outcome gave skeptics even more reason to be critical of the methodology. (How can it be any good if it has to be tweaked to yield predicable patterns?) This also raised further questions about the fairness of the calculation as well as its impact on Ohio’s accountability system.
The second major criticism around the value-added system is aimed at the Ohio Department of Education and the weight it allocates to the results in its overall ratings of schools. Some argue that the value-added results are given too much weight as it relates to both low performing and high performing schools.
Ohio has a complicated school report card for districts that can include up to 26 performance indicators based largely on achievement scores. However, meeting or exceeding value-added expectations plays a significant part in a school or district rating. A school district in Ohio can fail to have students in any grade or subject at the 75 percent proficiency level (the state goal) yet still receive a Continuous Improvement (C) rating if the district’s 4th through 8th grade students exceed expected growth on the value-added rating. The state gives a school district a bump of one full academic rating (from a D to a C, for example) for exceeding expected gains regardless of the overall proficiency results of students in the school.
Thus, schools may never make progress in actually closing the achievement gap in terms of student proficiency. Worse, critics contend, school districts with stagnant or even declining achievement scores are spared consequences because of their value-added gains. Meanwhile school districts with exceptional overall student achievement suffer consequences for not showing value-added gains. For instance, a district can lose its Excellent with Distinction (A+) rating if it fails to exceed expected gains regardless of the overall proficiency level of its students. There is a danger here of skewing rewards and consequences.
 |
The
various concerns and criticisms directed at Ohio's value-added system
have taken on a heightened sense of meaning as the state moves toward
teacher performance systems that mandate value-added results as a
significant component. |
|
 |
And yet another criticism of Ohio’s system relates to its usefulness to practitioners. Critics argue the whole process is invisible until after the student results are made public – months after they’ve moved on to the next grade and new teachers. Educators can’t use the data to make mid-course corrections during the school year or even to reflect on their performance before the next school year begins, as the results are issued annually in late August.
The various concerns and criticisms directed at Ohio’s value-added system have taken on a heightened sense of meaning as the state moves towards teacher performance systems that mandate value-added results as a significant component. As the stakes around value-added results go up for teachers, the scrutiny facing the system in Ohio will surely increase as well. Value-added is a powerful tool, but it needs to be used in a way that is both fair to its strengths and appreciative of its limitations. Ohio has the chance to be a leader in the use of value-added systems once again. This time it can show how to implement the lessons learned over the last four years to improve the transparency of the current system, while rethinking and adjusting the weight given to it when rating both schools and educators.
