Real time classroom observations by trained evaluators hold promise to accurately assess the quality of teaching and learning going on inside those four walls; an as-yet-untapped area of “education R&D”. But numerous roadblocks—lack of time and training, disagreement over rubrics, and teacher resistance among them—stand in the way of a systematized and smoothly-functioning assessment system. A group of researchers from Australia has developed a new framework that evaluates teacher and student engagement with course content that can be utilized while observing video-recorded lessons. A recent report explains their methods and describes the findings of its application.
The basis for the new teacher evaluation framework comes from the research of Michelene T. H. Chi, professor at the Mary Lou Fulton Teachers’ College at Arizona State University. Chi was a 2023 Yidan Prize winner for developing the ICAP (Interactive, Constructive, Active and Passive) theory of cognitive engagement to help understand how students learn.
A précis on ICAP’s four modes of engagement: “Passive” cognitive engagement is where learners are being oriented toward and receiving information from instructional materials, such as listening to a lecture or watching a video, without overtly doing anything observable related to more active learning. “Active” engagement requires focused attention while manipulating lesson materials or input—think underlining text or writing a summary of an essay. “Constructive” engagement refers to behavior that produces new ideas that go beyond the information given; students might relate new information to previous knowledge, generate inferences that are not explicitly stated in the text, or provide justifications that make the text or the problem solution more explicit. And “interactive” engagement is a group activity that meets two criteria: (a) the partners’ utterances must be primarily Constructive, and (b) the interaction must extend the generative nature of the prior contributions of the individual partners. In short, it’s a constructive-type engagement with a partner or two rather than the teacher.
Chi’s work argues three that there is a clear hierarchy among these four mode, with passive at the bottom and interactive at the top. Active engagement leads to stronger learning outcomes than passive, she says, because students engage with the material, relate it to prior knowledge, and store it in easily-retrievable ways. Constructive is better than active because students must provide explanations, raise critical questions, or complete other tasks that engage the highest levels of cognitive processing. And the interactive mode of engagement leads to the strongest learning outcomes because dialogue between learners can give rise to knowledge change processes and can create new knowledge that the partners could not have generated alone.
The Australian researchers devised an ICAP-based coding framework that distinguishes the four modes of cognitive engagement, provides reviewers clear operational criteria for defining the different modes when they observe them, and links the different modes to distinct learning outcomes. Best of all, the coding can be done using recorded lessons and transcripts, minimizing classroom disruption that can occur during live observation and maximizing the number observations a single reviewer can complete.
They watched and coded one thirty-minute video recorded lesson from twenty teachers from eight schools in the Greater Adelaide region of South Australia. All but one of the teachers taught in public schools, and all but one at the secondary level; each had more than five years of teaching experience. All had Bachelor degrees, and four also had master’s degrees. All of the teachers taught STEM-related subjects, including math, hard science (physics, biology, etc.), and soft science (psychology, health, etc.). Most schools were rated “less advantaged” via the socio-economic criteria used by the South Australian state government, including several landing at the most disadvantaged level (7 on a 1–7 scale). A few schools were on the “more advantaged” side of the spectrum, but no higher than a 4 on the scale.
The observers coded the lessons and tasks the teachers presented and assigned, as well as the verbal instructions they used in whole class directions, to determine which of the four areas of student cognitive engagement they utilized. Student actions, talk, and outputs were also examined to assess whether the actual mode of engagement that students displayed matched what was described in the teachers’ intentions for each lesson or task. Inter-rater reliability testing showed strong correspondence between different observers’ coding of lessons and tasks.
All together, the observers identified and coded seventy-six lessons and teacher-assigned tasks. Thirty-eight percent of those were active, 35 percent were passive, 13 percent were constructive, and 12 percent were interactive. Observed student engagement with the lessons and tasks—meaning the mode students were in, versus the mode teachers intended—was coded separately. But there was a high degree of correlation—except that students were observed in passive mode more often than any other, especially in the hard-science classes and in the less advantaged schools. In other words, some lessons and tasks designed to be active, constructive, or interactive instead produced passive student engagement.
The researchers deem all of this to be problematic for numerous reasons—including the possibility that it shows teachers who believe that engaging with difficult material is beyond the capabilities of certain students and are choosing to teach “downward” to an entire classroom based on that belief. However, their concerns are predicated on several questionable convictions. Such as that ICAP’s interaction hierarchy is gospel and that students cannot fully and properly understand, say, physics without a specific (and much higher than observed) percentage of constructive and interactive engagement with the material. Their basis for concern is also dependent on the observation of a single lesson, which is not able to account for the flow of a semester- or year-long course, no matter how accurate their coding instrument may be. Many such courses, especially in the hard sciences, typically start with introduction of new and unfamiliar material via lecture and progress to independent and group activities (experiments, reports, and the like) designed to demonstrate and reinforce the new concepts. For all we know, the very next lesson from those teachers, not observed or coded, would have flipped the engagement mode timing entirely around with a whole-class discussion or an entire class period of hands-on lab work.
It’s not a bad idea for teachers to understand how much time they spend in each mode—the observational framework does seem adequate to that purpose—and to provide advice and techniques for them to keep students meaningfully engaged as much as possible. But this research does not yet go far enough to actually prove how much passivity is occurring or whether it’s a problem no matter what that percentage turns out to be. More observation, coding (AI, anyone?), and analysis are minimum requirements.
SOURCE: Stella Vosniadou et al., “Using an extended ICAP-based coding guide as a framework for the analysis of classroom observations,” Teaching and Teacher Education (October 2023).
