Despite the expansion of computer-based testing in schools over the last decade—and ongoing concerns about negative impacts on student outcomes—research on the topic is scarcer than one might expect. One of the more recent rigorous studies (2018) covered Massachusetts’s administration the PARCC exam and found that students taking the test online tended to score lower.
Is that still the case? A newer study from a trio of economists at the University of South Carolina examines the rollout of computer-based testing (CBT) in the Palmetto State starting in 2015. While some of the methods are similar to the Massachusetts study, the researchers here use a better poverty measure (direct certification versus school-lunch status) and are able to track effects for three years versus two.
In South Carolina, elementary and middle school students began taking social studies and science tests online in 2015—ahead of a legislative mandate requiring them in 2016—while online test administration in math and English language arts (ELA) began in 2016. However, not all schools were able to meet the deadline, and many applied for waivers due to insufficient security, bandwidth issues, and so on. CBT was nearly universal in the state as of 2018 but required several years of phased rollouts, which were leveraged in the study’s design.
The research team used data including the share of online testing for each grade level for each school for each of four tests (science, ELA, math, and social studies), administrative school-level data, and information on the level of technology per school, such as broadband access and devices in use per teacher. In the first analysis, they observe pre- and post-transition patterns for all subjects but have the most data for science and social studies tests, which equates to four periods prior to the modality change and three after. Their second analysis controls for various school and student characteristics and allows them to examine impacts by subgroup.
Their key finding shows negative effects from CBT on average scores, ranging from a 0.02 standard deviation (SD) reduction in math to a 0.09 reduction in ELA in the first year of CBT adoption. Perhaps more important, the negative effects do not fade away in either of the subsequent years of the study. When looking at grade levels, however, the researchers find a negative effect for ELA and science in both elementary and middle grades, while math and social studies is negative only at the middle school level. Additional analyses continue to show that the impact on ELA is the most negative, and that middle school students in general experience more significant decreases in test scores than do elementary kids.
Next they look at differences by poverty status—defined as students in poor households receiving food stamps or other monetary benefits. Results reveal that CBT is apt to lower test scores for all subjects for students in poor households, especially in ELA. Among non-poor households, CBT has no significant effects on social studies and math, but negative impacts on science and ELA, although these effects are much smaller than the negative impacts on students in poor homes.
Finally, they find smaller negative effects of CBT in schools where technology is more readily available (measured by tech devices in use per teacher). Poor students still show negative effects from CBT even in schools with better technology, but those effects are more muted.
Obviously, the persistence of negative effects in this study is worrying, as one would hope the adoption of any new technology or procedure would get smoother over time. Analysts speculate that, because many students switched from elementary to middle school buildings between test administrations, they could easily have experienced test-mode whiplash, going from pencil to computer (or vice versa) more than once.
Still, other studies say that it is not the mode of test administration per se that’s the problem. It has to do with things like computer practice. To wit, it is more difficult for some young children to type than to handwrite their answers, especially if they have limited exposure to computer technology.
No matter what’s at play, the mechanics of staggered rollouts and pencil-and-paper alternatives to CBT in 2018 read like nostalgia in 2023. As many as forty-eight states now have computer-based tests (with some still also offering paper, too). Moreover, the pandemic forced students and teachers to rely on electronic teaching and learning. We know that this forced experiment led to dismal academic results and misery. But did some determined youngsters forge through the adversity and pick up more familiarity and facility with computers that perhaps stood them in good stead on state exams? One can hope.
SOURCE: John Gordanier, Orgul Ozturk, and Crystal Zhan, “Pencils Down? Computerized Testing and Student Achievement,” Education Finance and Policy (October 2022).
