“Nudges” to induce specific actions take many forms in education. For example: advertising about state aid to boost college enrollment, personalizing data to urge students to choose higher-quality colleges, targeting feedback on test performance to increase future scores, and offering small financial rewards to complete a series of tasks related to college applications. Impacts tend to be slightly positive, but the cost and administration of the interventions can outweigh their utility. Text-message-based nudges in particular hold much promise, especially if they can be automated and personalized to a degree. A recent working paper looks at a randomized control trial of “chatbots” to improve course performance in college, and it finds mostly encouraging results.
The study took place at Georgia State University (GSU) in an Introduction to American Government course, a high-enrollment course required for graduation. A full-time faculty member teaches each section of the course, and approximately 60 percent of GSU students attend sections that are online and asynchronous. In fall 2021, the researchers randomized over 500 students enrolled in the online version of the course to treatment or control groups. All received standard communications from the course instructor about upcoming due dates and suggestions to meet with the professor when they failed to turn in an assignment or exam. But students in the treatment group also received messages from the course-specific chatbot known as “POLS Pounce,” sibling to the GSU system-wide main bot (“Pounce,” named after the school mascot) that reminds all students of administrative matters.
POLS Pounce sent two to three customized messages each week via text, nudging treatment students to complete late assignments and inviting them to engage with the bot with questions. It also provided students with estimates on the time that each assignment would take so they could plan ahead, and encouraged them to do very specific tasks towards completion of the larger assignment each week. Messages included the student’s name and customized the communication based on what they did or didn’t turn in. Importantly, the teaching assistant for the course answered any questions the same day for anything that the AI technology was not programmed to address.
Analysts found that the academic chatbot significantly shifted students’ final grades, increasing the likelihood of receiving a course grade of B or higher by 8 percentage points relative to the control group. They also found large and significant treatment effects for first-generation students, as the intervention increased their final course grades by about 11 points on a 100-point scale (and a 16-percentage-point increase in earning grades of B or higher), as well as their completion of and performance on individual course deliverables, such as activities and exams. For instance, first-generation students were 13–15 percentage points more likely to submit their online exams on time and spent, on average, more than ninety additional minutes reading their digital textbook throughout the semester than did their control group peers. Analysts also saw a 10-percentage-point increase in earning a B among Black students. Finally, 77 percent of students reported that POLS Pounce was at least somewhat useful to their studies, and 92 percent recommended using it in the course again. Researchers also found that the students did not perform any worse in other classes due to increased time spent in the focal government course.
Analysts hypothesize that the chatbot was of particular help because it broke down course deliverables into definable tasks and urged students to address those tasks each week—as well as estimating how much time they would need to complete them. Research has shown that time management skills are crucial for academic success in college and could have been especially helpful to first-generation students who might not have gotten advice from their families about college study habits. Also a plus: the assurance of human help if the chatbot couldn’t address the students’ questions or concerns. We’re told that POLS Pounce is a low-cost nudge, but estimated expenses are not provided.
Clearly text reminders make sense in college, but since phones have become ubiquitous among thirteen- to seventeen-year-olds, as well, we should ponder nudging kids in middle and high schools, too. After all, they’ll be looking at their phones anyway!
SOURCE: Katharine Meyer et al., “Let’s Chat: Chatbot Nudging for Improved Course Performance,” working paper from Annenberg Institute at Brown University (April 2022).
