The Education Gadfly Weekly

Eleven weeks back, those of us at the Fordham Institute reported that current accountability systems in most states give primary and middle school educators scant reason to attend to the learning of high-achieving youngsters—which is to say, those systems generally fail to create incentives, rewards, or even transparency regarding the learning gains that schools are producing for students who have already crossed the proficiency threshold.

We coupled that bleak finding with a reminder that the Every Student Succeeds Act (ESSA) creates a rare opportunity for state leaders to rethink their accountability systems and thereby set matters right.

Now we’re back with a similar appraisal of state accountability regimes as they affect high schools. This one isn’t quite as gloomy, as we find more states paying attention to high achievers in the upper grades—and the structure of high school is more amenable to such attention, organized as it is around courses and course sequences, including electives that may include honors and AP classes and other means of accelerating one’s learning.

Not as gloomy, no, but not exactly rosy, as we can identify just four states that are doing it well today (Georgia, Ohio, Pennsylvania, and Texas) and four more (Alabama, Idaho, Louisiana, and New York) that are clearly moving in the right direction based on their recently released plans for holding schools accountable under ESSA.

As we found in the earlier grades, most states’ accountability systems for high schools lean heavily on proficiency rates—measuring the proportion of students who reach the proficient level on state tests, mostly in English and math. That’s not a great metric for school quality in the first place, considering how closely it correlates to student demographics and prior achievement rather than illumining the school’s true effectiveness as a learning engine. But it’s doubly lacking with respect to high achievers, as it signals to schools that those kids—who were already proficient on the first day of the school year—“aren’t your problem.” Why sweat teaching them more when the school gets no credit for doing so? (Fortunately for the kids, many right-thinking educators do pay attention to their students’ needs and opportunities, not just to state-level ratings and policy machinations.)

Accountability schemes for high schools have also focused heavily on boosting graduation rates. That’s an important thing to do but, again, does little for high achievers, nearly all of whom were already on track to graduate. Along the way, we must also note, the push to raise graduation rates has fostered such dubious practices as ersatz “credit-recovery” options for those who didn’t take or pass the requisite courses the first time around and who may therefore not get truly equivalent learning, even if they wind up with a diploma.

Nevertheless, we’re pleased to report some positive developments. For example, we found twenty-two states giving (or planning to give) high schools some accountability points for helping students earn college credits before graduation via AP, dual enrollment, and the International Baccalaureate (IB). Note that we only laud states that focus on the actual attainment of college credit during high school—for instance, rewarding schools where lots of kids pass AP tests (i.e., performance), not where they get a lot more kids simply to enroll in AP courses (i.e., access).

The impulse to get more students, especially poor and minority youngsters, into such advanced options is entirely commendable, but here, too, a worthy goal can have unintended side effects—in this case, by leading to the inclusion of students who aren’t actually prepared to succeed in more challenging academic settings. It’s not clear from the research literature that sitting in an advanced classroom but not succeeding in the course itself does a student much good. And one must also ask whether such an approach is good for the high-ability kids in those classrooms who truly are prepared to get the most from them. All too often, we sense, the well prepared pupils who end up forfeiting some of the benefit of such learning opportunities are themselves from disadvantaged backgrounds, as it is their schools—not the fancy high schools in posh suburbs—that tug hardest to open those classroom doors wider and push kids through them who may (through no fault of their own) not be up to the challenges within.

Again on the mostly positive side, we find thirty-two states that calculate—or intend to calculate—academic growth at the high school level using models that include high achievers. That does not, however, mean that they necessarily give sufficient emphasis to growth versus proficiency.

As is evident from the to-ing and fro-ing in the paragraphs above, the dark clouds we spotted on the high school horizon often have silver linings, just as the fluffy ones carry some threat of gloom. That’s simply the state of school accountability in the U.S. today. So yes, we see a positive overall trend, as a number of states begin to upgrade their accountability systems in ways favorable to high achievers. But—as demonstrated by the blunt fact that we can only confer overall high marks on eight states at this time—there is a long way to go.

It’s important for America’s future that we persevere in that journey, because our track record at the high end of academic achievement at the high school level has been seriously disappointing for far too long. Whatever modest gains we wrought in the early grades in the NCLB era, as gauged by measures such as the National Assessment of Educational Progress (NAEP), twelfth-grade scores have been flatlining for decades, especially at what NAEP terms an advanced level. The same is true of SAT and ACT scores. As for international metrics such as PISA and TIMSS, we’re being sorely outclassed by far too many other countries, both in the fraction of our young people who reach the upper ranks on those metrics and in the representation of lower-SES and minority youngsters (save for Asian Americans) among those who do make it.

Getting the accountability system right for high achievers will not, in and of itself, propel us into the top tier of high achievement on a global scale, but it’s a key component of such propulsion.

Fortunately, states now have an opportunity to put America’s schools on the right path. It will take leadership and courage, however, as naysayers will always insist that any attention given to high achievers is inherently elitist, if not classist or racist. These nattering nabobs of negativity are simply wrong. There are hundreds of thousands of American teenagers ready to work harder, reach higher, and go further, if only we give them the chance. Many are kids of color and come from poor families. They deserve our attention. State accountability systems can send strong signals about who matters. The right answer is everyone—including high achievers.

Those of us at the Fordham Institute have long held that there’s no one best way to design a state accountability system. It’s not just that we can’t even agree amongst ourselves about the relative importance of measuring student growth vs. student proficiency (though that’s true). It’s also because we understand that, as with all policy endeavors, this one amounts to a series of trade-offs. Perhaps there are some wrong answers (such as relying exclusively on proficiency rates in reading and math to judge school quality, or measuring school spending and other inputs and calling it “accountability”) but mostly there are a whole bunch of right and partly-right answers, depending on policymakers’ goals and states’ idiosyncrasies. That’s why, nine months ago, when we hosted our ESSA Accountability Design Competition, we intentionally decided not to award a “winner.”

Still, we know that states are now putting pen to paper on their accountability plans and that many of them want advice about what to do. So no more hesitating or prevaricating. Here’s our attempt—just David and Mike, mind you, not Fordham at large—to lay out an ideal accountability system for states. Consistent with the guidelines for the original accountability contest, our proposal includes design objectives, the indicators we would use to assign summative ratings to schools (should the feds in fact force states to do this), as well as recommendations for the U.S. Department of Education. 

Design Objectives

Our proposed accountability system has three goals:

1. To gauge school performance as fairly and accurately as possible (most importantly, by not penalizing schools for factors outside their control, such as students’ prior achievement)
2. To encourage schools to focus on all students’ long-term success
3. To foster an environment that rewards quality by empowering parents

As these goals suggest, we believe that ESSA provides an opportunity to move accountability beyond the constrained vision of No Child Left Behind—and that states should seize that opportunity. Rather than hold schools to account for student performance at one point in time, we can now hold them accountable for students’ progress from one year to the next. Rather than focus exclusively on ensuring that kids are minimally proficient, we can now reward improvement across the achievement spectrum. Rather than making state bureaucrats solely responsible for holding hundreds or thousands of schools to account, we can share this responsibility with those with the greatest stake in the final outcome: parents and other adult caregivers.

System for Rating Schools

Consistent with the Department of Education’s interpretation of ESSA, our proposed accountability system assigns summative school ratings based on a range of indicators, which we describe below. To be clear, we believe ESSA can be read to allow for multiple grades or ratings for schools (for instance, one each for Academic Achievement, Student Growth, Graduation Rates, Progress toward English Language Proficiency, and Other Indicators of School Quality and Student Success). That would be our preferred interpretation. But assuming the feds stick to their guns, here’s how we’d combine these factors into a summative grade (A-F).

Indicators of Academic Achievement (10-25 percent of summative school ratings)

In the average state, measures of academic achievement currently count for about half of schools’ summative ratings. However, because these measures are strongly correlated with student demographics and prior achievement, we believe they should count for at most a quarter of schools’ ratings going forward. Further, instead of using raw proficiency rates, which encourage schools to focus on the “bubble kids” (i.e., those just above or below the proficiency threshold) states should use average test scores (like Nebraska) or a “performance index” (like Ohio). Currently, at least seventeen states use (or plan to use) one of these methods, though it’s unclear whether the Department of Education will allow them to continue to do so under the new law.

Indicators of Student Growth (50-90 percent of elementary and middle school ratings, 40-80 percent of high school ratings)

Because they are the best indicators of schools’ overall performance, measures that capture the academic growth of all students should count for at least half of elementary and middle schools’ ratings (as they already do in six states) and at least 40 percent of high schools’ ratings (as they do in Michigan and New Mexico). Currently, forty-five states estimate growth in English Language Arts and math at the K-8 level, and thirty-five do so in high school. Thus, assigning more weight to these measures is something most states could do right away. In the medium term, states should also continue to develop their capacity to estimate growth at the high school level, as well as in other core subjects such as science and social studies, which will allow them to weight growth even more heavily in the future. Because growth scores can be unstable, even at the school level, we encourage states to average over two years (or three, if necessary) when calculating school grades. We also strongly urge states not to use “growth to proficiency” measures, as these encourage schools to ignore the needs of their high-achievers (and are poor indicators of school quality). Similarly, we urge those states that base a portion of their grade on the progress of low-achieving students, or other subgroups, not to overdo it. In our view, at least three quarters of whatever weight states assign to growth should be based on growth for all students.

Indicators of Progress toward English Language Proficiency (Variable)

We will leave the debates over how to best serve English Language Learners to those with expertise in this area. However, common sense suggests that the weight assigned to ELL measures should vary based on the percentage of a school’s students who are classified as ELL.

High School Graduation (10-25 percent)

Though we don’t have strong opinions about how states measure graduation, it’s important that they don’t assign too much weight to this indicator, lest they encourage schools to lower their standards for earning a diploma. In our view, basing 10-25 percent of high schools’ ratings on some combination of 4- and 5-year graduation rates is a reasonable approach.

Indicators of Student Success or School Quality (10-20 percent)

There is broad agreement that states’ current accountability systems are unfortunately dependent on standardized tests that cannot capture all the skills that students need to acquire, and that have sometimes encouraged teachers to engage in harmful “test prep.” Yet many of the alternatives to testing that have been proposed, while promising in theory, are problematic in practice. Consequently, though we support the goal of reducing the emphasis on testing, we encourage states to be deliberate in their approach. ESSA’s “school quality” indicator provides an opportunity to experiment cautiously with new indicators and approaches, but that does not mean every new idea is worth trying. (For example, we are wary of indicators that can easily be gamed, such as those based on teacher surveys.) Over time, the weight assigned to these indicators may grow beyond the parameters we specify here, but first we need to figure out what works.

For now, here are five ideas we believe states should consider:

“College and Career Ready” indicators: Many states already include AP, IB, ACT, and SAT achievement in their high school rating systems, and we heartily endorse all of these of these measures, especially those tied to achievement on AP/IB tests, which are precisely the sort of high-quality assessments that critics of dumbed-down standardized tests have long called for. Likewise, we support dual enrollment-based measures, provided there is some form of quality control (e.g., provided that the credits students earn are accepted by state universities). Finally, we endorse indicators that are tied to industry credentials or certificates, which can be useful to students who are entering the job market directly out of high school. New Mexico, which already includes more than a dozen “college and career readiness” indicators in its high school accountability system, is a good example of what is possible in this area.

Subsequent performance/persistence: How students fare after they leave a school says a lot about what they learned while they were enrolled, and the degree to which that learning was accurately reflected in their test scores – or not. To guard against illusory achievement gains, states should rate elementary and middle schools based on the on-time promotion rate of students in the next two grades after they leave a school (as Morgan Polikoff of the University of Southern California recommends). Similarly, they should rate high schools based on postsecondary remediation and/or completion rates, which are preferable to enrollment rates.

Student/teacher retention: In a choice-based system, student retention is an important indicator of school quality that disincentivizes “creaming.” Similarly, teacher retention is an important indicator of teacher satisfaction that is strongly correlated with student growth. Because they are essentially immune to gaming, both of these ideas deserve more attention than they have received to date. In neither case is 100 percent retention the goal, but very low retention rates are surely a sign that something is amiss.

Chronic Absenteeism: Because the link between attendance and students’ long-term success is so clear (and because most states already collect attendance data), chronic absenteeism is an obvious candidate for the “school quality” indicator. States might also consider including chronic absenteeism for teachers.

Student surveys: Many teacher evaluation systems already incorporate the results of student surveys, which research suggests can also predict school and principal value-added. Unlike teacher surveys, which are easily gamed, student surveys are a potentially useful addition to existing evaluation systems, provided that states take sensible steps to ensure the integrity of the results.

Obviously, none of these measures is perfect. In particular, because schools that serve difficult populations are likely to have higher student/teacher turnover, higher remediation rates, and lower attendance, these measures are likely to be biased if the goal of the system is to gauge school performance fairly. In light of this concern, depending on their goals, states may wish to adjust schools’ scores on these indicators by controlling for demographics, geography, and other factors, much as they already do when estimating student growth.

Recommendations for the U.S. Department of Education

We have three recommendations for the U.S. Department of Education, which is expected to release its final ESSA regulations soon.

1. Allow states to use a performance index as their measure of academic achievement.

Almost none of the participants in our ESSA design competition recommended that states use proficiency rates alone, reflecting the near-universal consensus that such rates are a bad measure of school quality.

2. Allow states to vary their approach to rating schools in reasonable ways.

As noted above, common sense suggests that the weight assigned to the ELL indicator should vary with the proportion of the student body classified as ELL. Similarly, because growth measures may do a poor job of capturing the progress of high-achieving students, some states may want the weights assigned to achievement and growth to vary based on the level at which a school’s students are achieving.

3. Let states decide whether they will assign summative ratings to schools

There is a case for summative school ratings, which send an unambiguous message about the quality of a school to parents who might otherwise be overwhelmed with information. But there is also a case against such ratings, which often obscure more than they illuminate by conflating fundamentally incommensurable indicators, such as growth and achievement. Consequently, states ought to decide for themselves whether to assign such ratings.

***

We are in one of those rare moments in the education world when real change is not only possible but likely. Beneath the hideous clamor of the presidential campaign, the quiet murmur of state boards of education and gubernatorial subcommittees speaks to survival of a different America, in which Democrats and Republicans can still come together to do what’s best for kids. Getting accountability right is important. Let’s not let them down.

On this week's podcast, Mike Petrilli, Alyssa Schwenk, and Brandon Wright discuss what President-elect Trump might mean for the Department of Education and school discipline. On the Research Minute, Amber Northern examines the effects of teaching assistants and nonteaching staff on student outcomes.

Amber's Research Minute

Many prior studies have found that low-income students have less qualified teachers based on measures such as years of teaching experience, teacher licensure test scores, certification status, and educational attainment, but they say very little about how these differences relate to closing the achievement gap, nor do they examine the magnitude of how differences in access to effective teachers might impact performance.

Yet a new Mathematica study is full of surprises. It examines low-income students’ access to effective teachers in grades four through eight over five years (2008–09 to 2012–13). “Low income” is defined as being eligible for free and reduced-price lunch (FRPL), and “high income” includes everyone else (so not much nuance there). The sample includes twenty-six geographically diverse, large school districts across the country, with a median enrollment of 70,000. And analysts measure the effectiveness of each teacher in the district using a value-added model.

There are five key findings.

First, contrary to conventional wisdom, teachers of low-income students are nearly as effective as teachers of high-income students on average (a difference of one percentile point). Specifically, the average teacher of a low-income student is just below the fiftieth percentile, while the average teacher of a high-income student is at the fifty-first percentile.

Second, high- and low-income kids have similar chances of being taught by the most and least effective teachers. For example, 10 percent of both high and low income kids are taught by one of the top 10 percent of teachers in a district.

Third, teachers hired into high poverty schools are equally effective as those hired into low poverty schools. Though both the new hires are less effective than the average teachers, and high poverty schools have more new hires than low poverty schools, neither makes much of a difference because those differences are already small and the performance of new hires improves fast: on average, they become as effective as the average teacher after one year.

Fourth, not surprisingly, on average, teachers who transfer to schools that are higher in poverty than the one they left are less effective than the average teacher. Yet those differences don’t impact equity much because just under 4 percent of all teachers transfer to schools in a higher or lower poverty category anyway (a little more than 4 percent move between schools with similar poverty rates).

Fifth and finally, teacher attrition doesn’t much affect access to effective teachers among high- and low-income kids because the leavers are equally effective among high- and low-poverty schools. Only in a small subset of districts (three out of twenty-six) did they find inequity in access to effective teachers—and it was in math only. In those three districts, if you provided high- and low-income kids with equally effective teachers from fourth to eighth grade, you’d see a reduction in the student achievement gap by at least a tenth of a standard deviation, which is equivalent to four percentile points over a five-year period.

With all that said, the sample the study uses is not nationally representative, even though it is geographically diverse and mostly includes large districts that are lower performing. It also mirrors the types of achievement gaps we see nationally, including in NAEP performance. Therefore, these findings may not hold in small districts or rural areas, for example.

Furthermore, it’s possible that the poorest children in the country (say, those at the tenth percentile of the income distribution) are in fact getting less-effective teachers than the richest kids (those at the ninetieth percentile, for example). But this study couldn’t examine that question because it relied on a binary definition of socio-economic status (i.e., whether a student was eligible for FRPL or not)—and again, findings are not nationally representative.

Still, analysts conclude with a simple summary: The achievement gap arises from factors other than students’ access to effective teachers.

Given that this bottom line finding is the result of an expensive study commissioned by a federal agency and conducted by a well-regarded research shop, it represents a big debunking of conventional wisdom.

SOURCE: Eric Isenberg et al., “Do Low-Income Students Have Equal Access to Effective Teachers? Evidence from 26 Districts,” Institute of Education Sciences, U.S. Department of Education (October 2016).

Recently, the New York City Independent Budget Office (IBO) released a brief that examines how various factors determine which high schools students prefer to attend.

In 2003–04, New York City public schools implemented a computer-based system that assigns middle school students to the city’s 691 high school programs. Students submit a list of up to twelve high school programs ranked in the order of preference; an algorithm that accounts for admission qualifications and seat availability assigns as many students as possible to their first-choice. The IBO used data from 2004–05 through 2011–12 to compare the programs students preferred to the graduation rates of those programs, student test scores, and the proficiency scores of NYC middle schools to see how school quality and student achievement influence student preferences.

Within the NYC public school system, there are seven types of high school programs a student can choose to attend. Each type varies in terms of how selective it is of the students it admits. Some, like fine arts programs, require students to audition before they are accepted. Others screen applicants by evaluating entrance essays, test scores, or attendance records. The least competitive programs usually admit students through a lottery where every applicant has an equal chance of being selected. The IBO found that there is much greater demand than seats available in the more selective high school programs, which regularly turn away qualified students.

The brief also looks at how individual student achievement affects which program students list as their first choice. Interestingly, the data show that lower-achieving students are admitted to their first-choice program at a higher rate than high-achieving students. Specifically, in 2011–12, 58 percent of lower-achieving students were admitted to their first choice, while only 53 percent of the highest-achieving students could say the same. This is due in part to the fact that the best students usually list programs with more competitive admissions processes as their first choice. Meanwhile, low-achieving students tend to gravitate toward those that have a relaxed screening process or no screening process at all. Consequently, they have a greater chance of being admitted to their first choice.

The IBO’s most surprising finding was the strong correlation between the performance level of the middle school a student attends and his or her high school preferences. Regardless of their individual performance level, students from lower-performing middle schools are less likely than students at higher-performing schools to select high-performing high school programs as their first choice. This suggests that high-performing students in mediocre middle schools seem less likely to even see the city’s best high schools as within their grasp, which is troubling. This finding remains consistent when applied to a variety of measures for school quality. For example, students from middle schools that perform well are more likely to select high schools with higher graduation rates than the high-achieving students from lower-performing middle schools.

Understanding New York City’s high school program selection process helps draw awareness to the divergences between the preferences of lower- and higher-achieving students. Although this report is limited in regards to providing specific information on every variable that may impact student choice, the information it gives on the impact of a small fraction of these variables should not be ignored. While the system is quite effective at assigning the majority of students to their first-choice school, it is discouraging that students from lower-performing middle schools are less likely to show interest in attending—and perhaps see themselves as qualified for—more selective and better quality programs. To counter this, New York City school districts should consider ways to make information about highly selective programs more accessible to students attending lower-performing middle schools. Providing such resources will hopefully lead to an increased interest in more competitive high school programs, particularly among high-achievers at struggling schools.

SOURCE: Przemyslaw Nowaczyk and Joydeep Roy, “Preferences and Outcomes: A Look at New York City’s Public High School Choice Process,” New York City Independent Budget Office (October 2016).