Steve Farkas is veteran public opinion researcher, co-founder of the FDR Group, and author of the Fordham Institute’s recent report, Yearning to Break Free: Ohio Superintendents Speak. The following was written in response to a review of the report by the Think Tank Review Project at the National Education Policy Center.
The FDR Group’s recent survey of Ohio’s school district superintendents, Yearning to Break Free (online here), conducted on behalf of the Thomas B. Fordham Institute, found these local education leaders eager to overhaul the collective bargaining process and to increase their authority over staff and money. “Give us autonomy,” they said, “and hold us accountable for getting results.” Easy to understand, right? Well, two unhappy University of Houston professors reviewing the study for the National Education Policy Center’s “Think Tank Review Project” had a lot of trouble comprehending it.
From the get-go in their review, the professors failed to realize that the study was giving voice to the opinions of these school leaders – not our opinions as researchers or even Fordham’s but those of Ohio superintendents. The professors say, “the authors of the study recommend [emphasis added] that ‘two promising ways to save districts money are to give superintendents greater control over combined state revenue streams and to mandate a statewide health insurance plan…’” But the report itself stated “Ohio’s superintendents think [emphasis added] two promising ways…” We (neither the FDR Group nor Fordham) didn’t recommend anything. Yearning to Break Free is a study of perceptions so the whole report is filled with such modifiers as “superintendents believe, say, think.” It’s hard to see how the professors missed that.
The professors had other troubles. They didn’t understand why we would force superintendents to choose between what they called “inappropriate dichotomies.” They balked when we asked superintendents what would be more likely to lead to improved student achievement: “significant expansion of management authority over staff” or “significant increases in school funding.” But the dichotomy is valid and interesting – and it pushes respondents to think harder and prioritize. (By the way, the result was 50 percent to 44 percent.) I’ve got a really unrealistic question for the professors: If you had to choose, would you rather have lunch with President Obama or Lady Gaga? See, even a far-fetched dichotomous question can tell you something interesting about your preferences.
I have my own complaints about what the professors say. The professors said the study aggregates responses for superintendents and charter leaders. That’s just plain wrong – it doesn’t. They talk about a poor response rate when the response rate was very high for a study of this type – 40 percent. Real-world survey researchers – myself included – dream of getting such high response rates on a regular basis. The professors say we don’t know how the superintendents who didn’t answer the survey differ from the people who did. Well, by definition that’s almost always the case. We made extraordinary efforts to get non-respondents to take the survey (see the appendix of the report for more on those efforts). That’s how we got such a high response rate.
Perhaps most amazing to me: in the end, after all the mistruths and mistaken comprehensions, the professors agree that the survey correctly captures the attitudes of superintendents! They say: “some aspects of the report are not surprising…it is entirely predictable that superintendents would like to have greater control over both teachers’ salaries and state regulations.” Oh. So this survey that they claim is badly designed, badly worded, not representative, biased, and narrow somehow got the right answer? Now that is interesting.
