Last week here in Flypaper, I critiqued the RAND Corporation for selectively emphasizing positive and de-emphasizing negative findings of its first-ever randomized control trial of restorative justice.
A few days later on the RAND Blog, the authors defended the presentation of their findings:
Although not mentioned by our critic, we did highlight the presence of negative findings in our reports. We emphasized negative findings in our research brief that was directed to the media, in the executive summary of our larger research report, and in the findings posted on our internet page.
If RAND were a conventional think tank, this matter would be of limited interest. But RAND has earned an august place for its neutral and disinterested analysis, and as such, policymakers are inclined to provide them with a greater presumption of bipartisan deference on a policy initiative such as restorative justice. It is, therefore, important to assess whether that deference is due in this case.
For the reader’s ready reference, here is a bullet-point list of the restorative justice study’s major findings, organized by their appearance in the executive summary/research brief of the RAND report:
1. “Strategies to build program were successful”
A. Teachers reported using affective statements, circles, or conferences
B. High school staff did so less than elementary school staff
2. “Teachers observed that restorative practices improved school climate”
A. Teachers reported better teacher leadership and school leadership
B. Teachers reported better teaching and learning conditions
C. Teachers reported better classroom management
3. “Average suspension rates and disparities in suspension rates were reduced in the restorative practices schools”
A. Suspension rates declined in elementary schools
B. Fewer students were repeatedly suspended
C. Fewer students were transferred to alternative schools
D. Suspension rates for black students, low-income students, and females declined
4. “Restorative Practices did not positively affect all students in the treatment schools”
A. Academic outcomes worsened in middle schools
B. No suspension decline in middle schools
C. No decline in suspension for males, special education students
D. No decline in violence or arrests
5. Findings not mentioned in research brief
A. Academic harm to black students and students at predominantly black schools
B. Student reported deterioration in teacher classroom management
C. Student reported worse peer relationships
It seems noteworthy that, whereas the researchers made a point to disaggregate the suspension findings to point out that suspensions decreased for black students, they did not do the same with academic achievement and point out the decline in black student achievement.
The RAND authors explain the omission by contending that academic effects were of secondary interest compared to suspension effects, and that they had to exercise reasonable discretion in presenting findings spanning “seven outcome domains” in a short research brief.
It is, however, noteworthy that the only outcome domain completely omitted from the research brief was student perceptions, i.e., the fact that students believed that peer-to-peer relations deteriorated and that students believed their teachers’ classroom management skills became worse.
For their part, the researchers earnestly believe that their presentation was fair and balanced.
Readers are encouraged to exercise their own informed judgment.
But all close readers should find the call to action in the last line of the authors’ rebuttal revealing: “We trust that these public servants will join us in the effort to innovate and monitor these programs so that more children can stay in school and all children have the opportunity to learn.”
By calling on policymakers to “join” them in their “effort to innovate and monitor,” it’s clear that, on this issue, RAND sees its role as both promoter and evaluator. Think tanks such as the Center for American Progress on the left and the Heritage Foundation on the right have traditionally assumed a similar dual role. But historically, RAND has earned a position of unique respect for standing above the policy promotion fray by offering disinterested evaluation.
But the authors wear their ideological priors on their sleeves when they define the goal as ensuring “that more children can stay in school.” Such language is adopted straight from the world of advocacy. A more neutral way to express to that policy goal could be “decrease the percentage of students who lose a handful of days of classroom instruction because of bad behavior.”
Other worthwhile policy goals include increasing student achievement and improving classroom climate in the eyes of students. Some might even argue that disciplinary practices are a subordinate means to achieve those primary ends.
It’s disappointing that the RAND research brief omitted those negative findings.
But given the revealing defense offered by the authors, it is unfortunately not at all surprising.
