Add this report by the Center For American Progress (CAP) to the mountain of evidence disproving the notion in American public education that more money is incontrovertibly better (or even necessary to achieve greater productivity). And add it to the slowly growing pile of work that attempts to measure educational productivity, or “return on investment” – a concept CAP w
Add this report by the Center For American Progress (CAP) to the mountain of evidence disproving the notion in American public education that more money is incontrovertibly better (or even necessary to achieve greater productivity). And add it to the slowly growing pile of work that attempts to measure educational productivity, or “return on investment” – a concept CAP warns does not signify their endorsement of “unfettered market-based reforms” or a call to spend less on K-12 education.
Rather, after decades of constant growth in education spending with no real corresponding gains in achievement, it’s time to look at the bang we’re getting for all those bucks. The austere fiscal climate facing most states lends urgency to the task as policy makers and lawmakers figure out what to cut, what to preserve, and how to maximize scarce public resources. This study shows that many districts could actually boost student achievement significantly without spending a dime.
In the most comprehensive examination of public school productivity seen yet (it looks at 9,000 districts in 45 states), CAP not only measures productivity, or the “academic achievement a school district produces relative to its educational spending, while controlling for factors outside a district’s control,” but also hopes to inspire districts and states to undertake similar efforts to think about ROI, collect and make transparent such data, and put systems in place that will identify and sustain educational efficiencies. Productivity is defined in three ways (and with a helpful interactive map, one can look up any district in the study and see how in rates according to each).
Basic Return on Investment index: This rates districts on how much achievement they get for each dollar they spend – relative to other districts in the state. It controls for cost-of-living differences, as well as demographic variables like percentage of students who are low-income, limited English proficient, or special education.
How Ohio fares: There are over 285,000 Ohio students attending districts in the very lowest category of ROI. Many of these highly inefficient Ohio districts also spend a lot per-pupil compared to their peer districts. Large urban districts, as well as suburban and rural districts of all sizes, fall into this tier.
Adjusted Return on Investment index: This applies a similar approach as the Basic ROI, but uses regression analysis to control for variables and is possibly more precise than the Basic index, which may not be “sensitive enough to account for spending differences within states.”
How Ohio fares: Results for the Buckeye State are even worse than under the previous index , with almost 322,000 children attending districts that are the most inefficient. Several large and small city districts – like Akron and Mansfield – appear for a second time in this tier, as do rural and suburban districts of various sizes.
Predicted Efficiency index: This index accounts for a district’s per-pupil spending and demographic variables and then predicts how that district’s students should achieve; the index shows whether achievement is higher or lower than that prediction.
How Ohio fares: Interestingly, more Ohio districts do poorly according to this index than the others, with 406,000 students in districts at the bottom of the productivity barrel.
Nationwide, there are over 400 school districts –serving about 3 percent of total student population – rating as highly inefficient (appearing in the lowest tier on all three metrics). And low levels of productivity cost the nation as much as $175 billion a year.
In Ohio, four districts appeared in the lowest tier on all three rankings: Cincinnati, Cleveland, Columbus, and Youngstown. Over 150,000 students, or 8.7 percent of the state’s student population, attend a district on CAP’s most inefficient list. That means the number of kids attending a highly inefficient district is more than double the number nationwide. And regardless of which productivity index used, there are anywhere from 385,000 to 406,000 kids in inefficient districts (16 to 23 percent of the total public school population).
Not all the news is bad. Some districts consistently appear in the top tier of productivity, such as Perry Local School District (suburban) or Marlington Local School District (rural). CAP finds similarities between highly efficient districts, such as: smart use of data; a focus on outcomes (not inputs); and “willingness to make tough choices.”
Despite some important caveats about the data (CAP reminds us that the study’s methodology could not possibly capture every variable related to spending and achievement, and warns about the reliability of self-reported district data), the warehouse of information is well worth perusing. Finally, Ohio should heed the recommendation to follow in the footsteps of Florida and Texas, and produce school-level productivity measures with which to hold schools accountable and inform taxpayers of their return on investment.
"Return
on Educational Investment: A district-by-district evaluation of US educational
productivity"
Center for American Progress
Ulrich Boser
January 19, 2011
The National Association for Charter School Authorizers (NACSA) just released its third annual survey of charter school authorizers (a.k.a. “sponsors” in Ohio). To anyone interested in improving the quality of charter schools as well as outcomes for kids served by them, findings about charter school authorizing matter immensely.
First, more school districts are undertaking the work of authorizing – that is, monitoring and overseeing the education, fiscal, operational, and governance components of a charter school, and holding the school’s board accountable for performance. Between 2007-08 and 2010-11, 233 school districts became authorizers. Also of note, the survey found that charter agreements (i.e., the contract between the authorizer and the governing board of the charter school) with lengthier terms – for example, ten years as opposed to five years – resulted in a greater number of weak schools remaining open, largely because authorizers tend to close schools as a result of the renewal/non-renewal process, and focus on school evaluations at that time.
In terms of scale, large authorizers (those that authorize 10 or more schools) are more likely to implement authorizing best practices than their smaller counterparts. According to the report, roughly 700 authorizers oversee only one or two schools each, which in turn raises questions about whether they have the resources to effectively perform the work of authorizing. One finding that deserves additional study is that about one third of authorizers do not implement professional authorizing practices that relate to charter schools that contract with management companies – for profit or non-profit – that provide various services to the schools. This is a key issue, especially in situations where lack of capacity at the school governing board level has resulted in a power shift of sorts in which the management company potentially holds more power than the governing board of the school.
The 2010 survey (with the greatest response rate NACSA has seen yet) is an invaluable resource that offers insight into the core work that charter school authorizers perform.
"The State of Charter School Authorizing 2010"
National Association of Charter School Authorizers
January 2011
In this report, Dan Goldhaber of the Center for American Progress analyzes various teacher evaluation processes nationwide and provides recommendations for improving them, since current evaluation systems--to the surprise of no one-- fail to effectively measure the differences in teacher effectiveness. Value Added Measures (VAMs) can replace subjective, binary evaluation methods that have traditionally made teacher performance assessments exercises in futility.
Teachers become undifferentiated widgets under this binary evaluation system, despite evidence that some teachers are more effective than others. Goldhaber suggests that multi-year VAM estimates would decrease the risk of misclassifications by eliminating the statistical noise that can arise from having a strong class one year and a weaker class the next. Using VAMs would provide more transparency to potential classification errors because they are less subjective and can be easily evaluated since they are statistically based. He proposes an evaluation process that uses both observational measures and VAMs to identify low-performing teachers, and suggests that performance evaluations follow teachers from school-to-school and district-to-district.
However, Goldhaber notes the limits of VAMs as well. Most teachers, for example, are in classrooms or grades not covered by state standardized assessments, such as music, art, or first and second grades, making the use of VAMs impossible until state policies are changed. VAMs also fail to measure secondary functions of education, such as socialization behaviors that are taught in the classroom. Finally, VAMs focus heavily on student test scores, and this could result in the role of teachers being reduced to “teaching to the test.”
There is growing demand to reform teacher evaluation systems and value-added data plays an important role. In Ohio, Cincinnati Public Schools renegotiated contracts to establish a more rigorous evaluation process for their teachers. Evaluation methods will now include rigorous reviews by trained evaluators as well as measures of student growth. Statewide policy in Ohio also is headed down this path, as HB 21 which will require the use of value-added data in evaluating teachers (in grades and subjects for which it’s available) and principals for licensure.
"When the
Stakes Are High, Can We Rely on Value- Added? Exploring the Use of Value-Added
Models to Inform Teacher Workforce Decisions"
Center for American Progress
Dan Goldhaber
December 2010
Last Wednesday, the Ohio House Education Committee held hearings related to several education bills currently on the table, among them HB 21, which aims to lift the ironclad moratorium on virtual e-schools, grant a professional educator license to graduates of Teach For America wishing to teach in the Buckeye State, and require the use of student performance data in evaluating teachers. Fordham’s Terry Ryan, along with two teachers from Fordham-authorized charter schools, testified in support of the bill. (Read their testimonies here, here, and here.)
As we’ve noted before, this bill is a new iteration of Senate Bill 180 from the fall of 2009, for which Terry testified back then. What’s different this time around, however, is that a GOP-controlled House is likely to go further than the Democratic-controlled House of a year-and-a-half ago, and push for bolder changes in each of these areas (and others). While Fordham supports the provisions of HB 21, the legislature can, and should, go further that merely granting licensure to TFA alums, lifting e-school caps, and requiring value-added data. As Terry said:
…much has changed just in the last year and change is happening fast in states across the country. The reality is that Ohio risks being leapfrogged by dozens of states in many crucial areas of education reform…. Thus, while I support House Bill 21 and its passage, I encourage this legislature to think more boldly. While the bill is a good start, it does not go far enough toward enacting the education reforms Ohio needs
Terry then went on to describe a complete overhaul of teacher personnel policies that would include changing the way we recruit, reward, evaluate, and dismiss teachers – and getting rid of seniority-based layoffs.
If anyone on the committee had doubts about the importance of teacher effectiveness, whether or not it’s possible to differentiate for it, and whether Ohio should rethink its treatment of Teach For America graduates, Abbey Kinson’s and Jenna Davis’s testimonies powerfully dispelled them.
Kinson, contrasting her first experience getting “evaluated” (by a principal on her cell phone for about 15 minutes) in a DC Public School classroom with the way she evaluates teachers at Columbus Collegiate Academy, illustrated that distinguishing highly effective teachers from the rest really isn’t elusive if the right systems and metrics are in place. And Kinson is one to talk about teacher quality: she more than doubled her students’ proficiency in math last year and 100 percent of her seventh graders were proficient on Ohio’s math test. Despite this, Kinson laments:
I’ve proven that I know the best practices and have the skills to inspire kids to reach their potential. Yet, according to the Ohio Department of Education, I was not qualified to receive full licensure in middle school mathematics.
My experiences are typical of other Teach For America alums wanting to return to Ohio…. In many cases, great teachers have chosen to leave the profession and move on to higher paying jobs with a lot less red tape. Teach For America alums haven’t been welcomed back home to Ohio, but rather, obstacles have been put in our paths.
Davis, despite having a traditional educator’s background (in middle childhood education with concentrations in math and science), was still blocked from licensure in Ohio and told she’d be receiving a long-term sub’s license, despite the fact that she previously:
….Led [her] special education students to pass their Biology state test at a higher percentage rate than their counterparts in the state, and… led [her] Physical Science students to outperform their regular education peers on their state examination.
We’re hopeful that these testimonies not only compel lawmakers to pass the simple fixes in HB 21 but also to think more broadly and boldly about overall reforms to teacher evaluations, e-schools, and more.
Ohio has long awarded districts five calamity days – school days that can be missed and not made up due to inclement weather, power outages, or other catastrophic incidents. In 2009, Governor Strickland argued – as one of the stronger components of his education reform package – that Ohio in fact needed to add days to the school year to catch up with our better-performing peers nationally and internationally. The legislature balked at finding the funding to add more days to the school year and compromised with the governor by reducing the number of calamity days from five to three, in effect lengthening the school year at no cost. Strickland’s intention was to continue reducing the number of calamity days down to zero and then begin lengthening the school year after that.
What a difference an $8 billion budget hole, an election, and an unusually snowy winter, makes.
Districts were plowing through snow days in mid-December, an oddity for most of Ohio, and by mid-January many districts, especially rural ones, had used up their allotted days and were rearranging the remaining school year calendar to make up the time missed. When the new General Assembly got underway last month, State Representatives John Carey (R-Wellston) and Casey Kozlowski (R-Ashtabula County) introduced House Bill 36, which would increase from three to five the number of calamity days school districts aren’t required to make up and give districts additional flexibility on how they made up days after those five.
The bill had its first hearing last week before the Ohio House education committee (it will have a second hearing this evening and a companion bill is being considered in the Senate), during which its sponsors testified that granting more snow days to schools is a matter of safety. Committee members agreed that students and school employees shouldn’t be traveling to school in unsafe conditions. But lawmakers on both sides of the aisle made strong cases for requiring districts to make up time missed and no solid arguments in favor of the snow days were presented.
Rep. Ronald Gerberry (R-Youngstown), whose legislative district borders western Pennsylvania, pointed out that the weather patterns and road conditions on his side of the border are the same as those in nearby communities in the Keystone State, yet school kids in Pennsylvania make up every day of school that they miss.
Rep. Clayton Luckie (D-Dayton) commented that most Ohio students need more time in the classroom, not less, and questioned the academic impact of allowing more calamity days. “We’re already behind, why take away more class time?” he asked Reps. Carey and Kozlowski. Ironically, Wednesday’s committee discussion about the bill followed an update from State Superintendent Deb Delisle on efforts to raise the rigor of Ohio’s standards and model curricula in order for Ohio to become a national and international leader in academic performance. During that discussion the point was made that Ohio has a shorter school year than many other states and certainly most top-performing countries.
And several lawmakers inquired about the fiscal impact of snow days. After all, the state still pays districts for those days, and teachers and most staff still receive paychecks for them, even though they aren’t actually working. Reps. Carey and Kozlowski didn’t have an answer to the monetary cost of snow days, but rough estimates are easy to calculate.
Based on 2009 data (the most recent year for which numbers are available) of average teacher salaries and the number of teachers working in districts, one can calculate that the Columbus City Schools spends nearly $1 million in regular classroom teachers’ salaries and benefits alone for each day that school is closed and not made up later in the year.
The Little Miami School District outside Cincinnati is one of ten Ohio districts in “fiscal emergency” and its situation is so dire that district officials are seriously considering dissolving the district altogether. That district spends about $59,000 on teacher salaries per snow day. If the calamity day bill passes, the district will have spent more than $290,000 that it certainly doesn’t have to spare to pay teachers for days they haven’t taught.
Across the Ohio 8 urban districts, more than $18.5 million would be spent on just teacher salaries if the districts all used five snow days annually. And these numbers just reflect the cost of regular teachers – not principals, administrators, or other staff members who would be off work but paid for snow days.
Nevertheless, despite weak arguments in favor of the bill and while more important education matters are left untended, relieving districts’ snow-day pain has the apparent support of the governor and legislative leadership and seems headed for swift passage.
The recently released 2009 NAEP science results provide an in-depth look at how knowledgeable students across the country are in various science-related academic areas. The framework for 2009 sought to align the assessment with new developments in science standards, content, and curricula. The framework for the test broke down the questions into three separate areas: physical science, life science, and earth and space science. NAEP tests students in grades 4, 8, and 12.
The good news is that Ohio students outperformed students in many peer states. The average score of fourth-grade students in Ohio was 157 out of 300, which surpassed the national average of 149. Furthermore, 41 percent of Ohio fourth-grade students scored at or above proficient compared to the national average of 31 percent. The average score of eighth-grade students was 158 with 37 percent of students scoring at or above proficient, compared to national average of 149 (scale score) and 29 percent of eighth graders scoring at or above proficient. Ohio scored higher than 27 states and only four states scored higher (the remainder of the states were not statistically different than Ohio).
The bad news is that proficiency rates in both grades are still alarmingly low, and haven’t budged much in a decade (much like in reading and math). Chart 1 below depicts the proficiency rates of Ohio students in science over the past several years. Fourth-grade results have actually increased slightly over the last decade, but eighth-grade results have remained relatively constant.
The most recent NAEP science results also illustrate the “proficiency illusion” between different assessments. The Ohio Achievement Assessment (OAA) shows far higher rates of proficiency for the same Ohio students than does NAEP. Chart 2 below compares proficiency rates of the OAA and the NAEP for eighth-grade students in Ohio. Sixty-five percent of eighth-grade students scored at or above the proficiency level on the OAA, compared to the only 37 percent of eighth graders at the same level on the NAEP. The almost 30-point discrepancy between the NAEP and OAA demonstrates the difference in rigor of the two assessments.
This difference justifies a call for Ohio to create stronger assessment standards that more directly line up with national tests. To Ohio’s credit, in June 2010 the State Board of Education agreed to adopt the Common Core Standards in ELA and math. It will be imperative that they follow suit and adopt similar assessments in science and other subjects.
In conclusion the results of the NAEP science test are a familiar good news-bad news scenario. While it’s encouraging that Ohio outperformed most other states in science, proficiency in tested science skills are still very low (not to mention that as a nation we are well behind our international counterparts). These results of this test are yet just another confirmation to the same story that has been told for years. The U.S. cannot be competitive in a global economy that is dependent more and more on science and technology if we continue to produce lackluster results. Neither can Ohio.
The National Association for Charter School Authorizers (NACSA) just released its third annual survey of charter school authorizers (a.k.a. “sponsors” in Ohio). To anyone interested in improving the quality of charter schools as well as outcomes for kids served by them, findings about charter school authorizing matter immensely.
First, more school districts are undertaking the work of authorizing – that is, monitoring and overseeing the education, fiscal, operational, and governance components of a charter school, and holding the school’s board accountable for performance. Between 2007-08 and 2010-11, 233 school districts became authorizers. Also of note, the survey found that charter agreements (i.e., the contract between the authorizer and the governing board of the charter school) with lengthier terms – for example, ten years as opposed to five years – resulted in a greater number of weak schools remaining open, largely because authorizers tend to close schools as a result of the renewal/non-renewal process, and focus on school evaluations at that time.
In terms of scale, large authorizers (those that authorize 10 or more schools) are more likely to implement authorizing best practices than their smaller counterparts. According to the report, roughly 700 authorizers oversee only one or two schools each, which in turn raises questions about whether they have the resources to effectively perform the work of authorizing. One finding that deserves additional study is that about one third of authorizers do not implement professional authorizing practices that relate to charter schools that contract with management companies – for profit or non-profit – that provide various services to the schools. This is a key issue, especially in situations where lack of capacity at the school governing board level has resulted in a power shift of sorts in which the management company potentially holds more power than the governing board of the school.
The 2010 survey (with the greatest response rate NACSA has seen yet) is an invaluable resource that offers insight into the core work that charter school authorizers perform.
"The State of Charter School Authorizing 2010"
National Association of Charter School Authorizers
January 2011
In this report, Dan Goldhaber of the Center for American Progress analyzes various teacher evaluation processes nationwide and provides recommendations for improving them, since current evaluation systems--to the surprise of no one-- fail to effectively measure the differences in teacher effectiveness. Value Added Measures (VAMs) can replace subjective, binary evaluation methods that have traditionally made teacher performance assessments exercises in futility.
Teachers become undifferentiated widgets under this binary evaluation system, despite evidence that some teachers are more effective than others. Goldhaber suggests that multi-year VAM estimates would decrease the risk of misclassifications by eliminating the statistical noise that can arise from having a strong class one year and a weaker class the next. Using VAMs would provide more transparency to potential classification errors because they are less subjective and can be easily evaluated since they are statistically based. He proposes an evaluation process that uses both observational measures and VAMs to identify low-performing teachers, and suggests that performance evaluations follow teachers from school-to-school and district-to-district.
However, Goldhaber notes the limits of VAMs as well. Most teachers, for example, are in classrooms or grades not covered by state standardized assessments, such as music, art, or first and second grades, making the use of VAMs impossible until state policies are changed. VAMs also fail to measure secondary functions of education, such as socialization behaviors that are taught in the classroom. Finally, VAMs focus heavily on student test scores, and this could result in the role of teachers being reduced to “teaching to the test.”
There is growing demand to reform teacher evaluation systems and value-added data plays an important role. In Ohio, Cincinnati Public Schools renegotiated contracts to establish a more rigorous evaluation process for their teachers. Evaluation methods will now include rigorous reviews by trained evaluators as well as measures of student growth. Statewide policy in Ohio also is headed down this path, as HB 21 which will require the use of value-added data in evaluating teachers (in grades and subjects for which it’s available) and principals for licensure.
"When the
Stakes Are High, Can We Rely on Value- Added? Exploring the Use of Value-Added
Models to Inform Teacher Workforce Decisions"
Center for American Progress
Dan Goldhaber
December 2010
Add this report by the Center For American Progress (CAP) to the mountain of evidence disproving the notion in American public education that more money is incontrovertibly better (or even necessary to achieve greater productivity). And add it to the slowly growing pile of work that attempts to measure educational productivity, or “return on investment” – a concept CAP warns does not signify their endorsement of “unfettered market-based reforms” or a call to spend less on K-12 education.
Rather, after decades of constant growth in education spending with no real corresponding gains in achievement, it’s time to look at the bang we’re getting for all those bucks. The austere fiscal climate facing most states lends urgency to the task as policy makers and lawmakers figure out what to cut, what to preserve, and how to maximize scarce public resources. This study shows that many districts could actually boost student achievement significantly without spending a dime.
In the most comprehensive examination of public school productivity seen yet (it looks at 9,000 districts in 45 states), CAP not only measures productivity, or the “academic achievement a school district produces relative to its educational spending, while controlling for factors outside a district’s control,” but also hopes to inspire districts and states to undertake similar efforts to think about ROI, collect and make transparent such data, and put systems in place that will identify and sustain educational efficiencies. Productivity is defined in three ways (and with a helpful interactive map, one can look up any district in the study and see how in rates according to each).
Basic Return on Investment index: This rates districts on how much achievement they get for each dollar they spend – relative to other districts in the state. It controls for cost-of-living differences, as well as demographic variables like percentage of students who are low-income, limited English proficient, or special education.
How Ohio fares: There are over 285,000 Ohio students attending districts in the very lowest category of ROI. Many of these highly inefficient Ohio districts also spend a lot per-pupil compared to their peer districts. Large urban districts, as well as suburban and rural districts of all sizes, fall into this tier.
Adjusted Return on Investment index: This applies a similar approach as the Basic ROI, but uses regression analysis to control for variables and is possibly more precise than the Basic index, which may not be “sensitive enough to account for spending differences within states.”
How Ohio fares: Results for the Buckeye State are even worse than under the previous index , with almost 322,000 children attending districts that are the most inefficient. Several large and small city districts – like Akron and Mansfield – appear for a second time in this tier, as do rural and suburban districts of various sizes.
Predicted Efficiency index: This index accounts for a district’s per-pupil spending and demographic variables and then predicts how that district’s students should achieve; the index shows whether achievement is higher or lower than that prediction.
How Ohio fares: Interestingly, more Ohio districts do poorly according to this index than the others, with 406,000 students in districts at the bottom of the productivity barrel.
Nationwide, there are over 400 school districts –serving about 3 percent of total student population – rating as highly inefficient (appearing in the lowest tier on all three metrics). And low levels of productivity cost the nation as much as $175 billion a year.
In Ohio, four districts appeared in the lowest tier on all three rankings: Cincinnati, Cleveland, Columbus, and Youngstown. Over 150,000 students, or 8.7 percent of the state’s student population, attend a district on CAP’s most inefficient list. That means the number of kids attending a highly inefficient district is more than double the number nationwide. And regardless of which productivity index used, there are anywhere from 385,000 to 406,000 kids in inefficient districts (16 to 23 percent of the total public school population).
Not all the news is bad. Some districts consistently appear in the top tier of productivity, such as Perry Local School District (suburban) or Marlington Local School District (rural). CAP finds similarities between highly efficient districts, such as: smart use of data; a focus on outcomes (not inputs); and “willingness to make tough choices.”
Despite some important caveats about the data (CAP reminds us that the study’s methodology could not possibly capture every variable related to spending and achievement, and warns about the reliability of self-reported district data), the warehouse of information is well worth perusing. Finally, Ohio should heed the recommendation to follow in the footsteps of Florida and Texas, and produce school-level productivity measures with which to hold schools accountable and inform taxpayers of their return on investment.
"Return
on Educational Investment: A district-by-district evaluation of US educational
productivity"
Center for American Progress
Ulrich Boser
January 19, 2011
