On January 8, Indiana became one of five states singled out by the U.S. Department of Education for early approval of its No Child Left Behind accountability plan. These states were depicted as leaders that had set aside excuses and committed themselves to educating all students. Seven short months later, Indiana can, indeed, be characterized as a leader; but it is that state's role in obfuscating NCLB's goals that most deserves recognition.
In an earlier column [see http://www.edexcellence.net/gadfly/issue.cfm?issue=9#367], Checker Finn noted the acceleration of annual benchmarks at the core of Indiana's plan. For the next eight years, Hoosier schools will need, on average, to improve their pass rates on the state's test by seven percentage points every three years; but in the subsequent four years, seven percent gains are expected every single year.
Finn likened this to a "balloon mortgage" that will bequeath half of the state's expected improvement to a future generation of state and local education leaders. Such a balloon is especially unrealistic in a state like Indiana, where state test scores have shown no absolute improvement over the last 12 years.
Yet the "balloon mortgage" may not be the worst of Indiana's NCLB games. A more vexing obfuscation can be found in the state's treatment of expected pass rates on its tests. By applying an inflated "confidence test," state officials have substantially lowered performance expectations and assured that Indiana schools will never be held fully accountable for the 100 percent pass rate touted by NCLB.
Confidence tests are a common tool used in surveys and polls. In brief, they allow the results of a smaller sample of people to be extrapolated, with some degree of confidence, to the broader population. Since academic testing results are considered by many to provide only snapshots or estimates of actual student performance, confidence tests also may be used to identify a range of scores within which true performance actually falls.
Thus, the confidence test provides a "window" around the sample results to represent the area where real results are likely to be found. In public opinion polling, this window is indicated by the "margin of error" that is typically reported with polling data. If, for example, a poll finds that 55 percent of people support the president, and the margin of error--or window--is plus or minus three points, with a confidence level of 95 percent, that means we can be 95 percent confident that 52 to 58 percent of the population supports the president. Of course, wider windows create a higher degree of confidence--accompanied by less precision.
In standardized testing, margins of error are discussed in terms of standard deviations. Statistically speaking, a margin of error ranging one standard deviation above and below the sample results would provide a 68 percent degree of confidence that the actual results are found within that window. A window of two standard deviations would provide a 95 percent degree of confidence while three would yield a 99 percent degree of confidence.
In other words, the bigger the target, the more confident you can be of hitting it. The first of these options, one standard deviation, is quite common in statistical reporting. The second option, two standard deviations, is sometimes suggested for high stakes situations--like NCLB. The third option is extremely rare; yet that is what Indiana education officials opted for.
The results of this decision will have a huge impact on expected pass rates. The baseline pass rates reported this year to federal officials are 57.1 percent in math and 58.8 percent in English/language arts. But due to the large windows created by Indiana's 99 percent confidence test, the smallest groups of Indiana students--30 to 34--will need to reach just 36.7 percent passing levels in math and 40 percent in English/language arts in order to be said to have reached the (current) target. No group of students, no matter how large, will be required to achieve higher than 51.0 percent passing in math or 52.5 percent passing on English/language arts
These pass rates are substantially lower than those reported to the feds; and they also suggest that no school in Indiana, nor any subgroup, will ever be required to reach 100 percent passing--the ultimate goal touted by No Child Left Behind. That target will be lowered to the sill of a window that is three standard deviations wide--just as these initial benchmarks have been reduced.
Even the Council of Chief State School Officers--an organization that, until recently, was headed by Indiana's state superintendent--used only a 68 percent confidence test in its NCLB guidance for states and districts, while suggesting that states might consider the 95 percent test due to the high stakes nature of NCLB decisions. A 99 percent test, it seems, was not even contemplated. The chiefs' group also points to other statistical considerations such as minimum group sizes and multi-year averaging--also adopted by Indiana--as factors that could render NCLB results as invalid when combined with too high a confidence test.
This week's release of Indiana's AYP results [see http://www.doe.state.in.us/reed/newsr/2003August/Gov-NCLB-TitleI.pdf] suggests that these validity questions are, indeed, beginning to emerge. Despite a widely touted toughening of the rigor of Indiana's academic standards, the number of Title I schools identified as needing improvement actually declined this year from 156 to 117. Indeed, one of the schools highlighted by the state for coming off the improvement list actually saw its pass rate fall this year, from 55 percent to 43 percent. A valid accountability system would not have allowed this to happen.
One could hope that Indiana's state officials did not realize the impact of their actions. But using admirable language like "99 percent confidence" provides an awfully convenient cover; and several other manipulations of NCLB goals create a pattern that seems anything but accidental.
As another example, Indiana officials have also redefined some of NCLB's timing requirements. Several states worked hard this year to adjust their testing schedules so that school choice and supplemental services would be offered to qualified parents the year after their schools fail to meet adequate yearly progress, as demanded in NCLB guidelines. But Indiana officials ignored that demand, which means such consequences will be deferred up to one full year for Hoosier youngsters.
The camouflage here is Indiana's fall testing schedule. Given in September, each test actually measures the progress that students made during the previous school year. The test given to 6th graders next month, for example, will actually measure last year's progress against 5th grade standards, the 8th grade test will measure progress against 7th grade standards, and so on.
The results of these tests are returned mid-school year. Thus, Indiana officials will allow schools to delay providing mandatory supplemental services by one-half year and school choice by one full year. In other words, a school that fails repeatedly to make sufficient progress for a specific group of 5th graders will not be required to provide choice until those same students are in 7th grade.
Indiana officials have also played fast and loose with the definition of "dropout." After promising the feds in January that lawmakers would fix the state's flawed graduation reporting, the legislature actually worsened the problem by excluding from the dropout definition "students who have left school and whose location cannot be determined." That would seem to omit an awfully large percentage of what most people think of as dropouts.
There are other issues with Indiana's NCLB implementation, some of them shared with other states. For example, Indiana has no inter-district public school choice law--despite federal warnings that NCLB compliance would likely require such flexibility. Also, Indiana has declared that subgroups of less than 40 pupils need not abide by the law's requirement that 95 percent of all students must take the state tests, thus allowing substantial gamesmanship in the exclusion of students. And like lots of other states, Indiana has interpreted the phrase "persistently dangerous school" in such a ludicrously narrow way that no school is likely ever to get that label. (For California's similar gambit, see http://www.edexcellence.net/gadfly/issue.cfm?issue=26#65.)
Federal officials may well have been unaware of these games when they spotlighted Indiana as a model state. Indeed, among the issues noted above, only the "balloon mortgage" was described explicitly in federal application materials. Other games have surfaced in locally distributed documents or were cloaked in vague terms.
Yet this handful of examples from a single state must raise questions about other games that might emerge from states attempting to avoid new accountability measures under NCLB. Observers would be wise to keep careful watch.
Derek Redelman is a senior fellow and Director of Education Policy at Hudson Institute in Indianapolis. He can be reached at [email protected].
