In this research brief, Angela Duckworth of the University of Pennsylvania and David Scott Yeager of the University of Texas compare three measures of students’ non-cognitive skills: student surveys (in which students self-report on their non-cognitive skills), teacher surveys (in which the teacher provides his or her assessment of a student’s skills), and so-called “performance tasks” (such as the famous "marshmallow test"). After comparing these measures, the authors discuss their suitability for various purposes, including individual diagnosis, improved practice, program evaluation, and accountability.
According to the authors, each measure has advantages and disadvantages. For example, although student and teacher surveys are cheap and reliable, they suffer from “reference bias,” which occurs when individuals or groups use different frames of reference in making a judgment. Consequently, schools that are best at promoting non-cognitive skills may score lowest on a survey measuring such skills.
Unlike surveys, performance tasks don’t rely on the subjective judgments of students or teachers. Yet they too have drawbacks. To be a valid measure of a non-cognitive skill, a performance task must be administered under carefully controlled conditions, which may be difficult to achieve at some schools. They are also expensive and time-consuming, with a single task taking as long as twenty minutes to administer.
Obviously, using any of these measures for accountability purposes introduces additional dangers. For example, teachers could give their students higher ratings than they deserve, and students could be coached to provide the “correct” responses when prompted. To combat these dangers, the authors argue against incorporating surveys of non-cognitive skills into accountability systems or program evaluation (although they have some hope that the surveys can improve teacher practice). As for performance tasks, Duckworth and Scott suggest that a carefully designed suite of tasks could be used in program evaluation if the tasks were brief and administered by computer—although even this possibility is largely theoretical.
As the authors note, existing measures of non-cognitive skills, which were designed by researchers for use in their research, are ill-suited to other purposes. As accountability tools, therefore, these measures simply aren’t ready for prime time. More fundamentally, despite the undeniable importance of non-cognitive skills for students’ long-term success, the inherent difficulty of accurately assessing the character of America’s youth—and the incalculable risks associated with attaching consequences to such an assessment—should give policymakers pause. After all, we already worry about teaching to the test. Do we really want to worry about teaching to the performance task?
SOURCE: Angela L. Duckworth and David Scott Yeager, "Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes," American Educational Research Association (May 2015).
