The bulk of commentary and school district policy relating to AI and education focuses almost exclusively on questions regarding cheating. What does it mean for a student—or an education columnist, for that matter—if a chatbot can write a ten-page essay in a matter of seconds? How can teachers assign any homework and know for certain who or what is actually completing it? These are real questions with which the education sector must wrestle.
But there are other risks of AI and thorny questions looming on the horizon that are worryingly overlooked. Student privacy is perhaps chief among them. This concern became real to me when I read the following theoretical example from renowned education scholar John Hattie:
You are a sixteen-year-old girl on your way to school. Just before you get to the school gates, you waver and decide you would rather go meet your friends at the mall. Your Robo-Coach has access to the messages you exchanged with your friends the previous evening, it can sense your elevated heart rate as you stop to consider the alternative course of action, and it notes from your GPS location and lack of speed that you have stopped at an unusual place. From the combined data, it assesses an 87 percent probability that you will not attend school, and it interjects with a timely and personalized nudge (in a soothing versus commanding tone; male versus female voice; using an emotional versus factual plea, etc.) to push you through the school gates.
Chronic absenteeism is a real crisis in education right now. Research has found that regular alerts sent to parents about missing assignments and absences can improve pass rates, attendance, and retention. But this technology relies on human operators. Regular updates are time intensive—parent phone calls that I’ve made have lasted an hour—when teachers, administrators, and office assistants have other pressing concerns.
AI could supercharge this basic intervention without any extra resources or time demands on already overburdened school staff. Imagine if thousands of potential dropouts received such messages or parents got a simple text every time their kid missed an assignment, failed a test, skipped a class, or received a detention. Even if such messages only influenced a small percentage of cases, across millions of students, that would be a substantial nudge in the right direction.
However—and here is where student privacy enters the equation—in order for this actually to work, any such software requires access to vast sums of private, personal data: messages, family history, GPS location, previous attendance records, and biometric and basic identifying information. AI software is only as good as the data on which it is trained. An essential infrastructure for AI to function well is information. If electricity needs power plants, AI needs data stores.
Modern large language models (LLMs) like Chat GPT, for example, are distinctly powerful because they have essentially read the entirety of the Internet. With trillions of examples of text, it is functionally a high-powered, predictive language machine. What word is likely to come next based on the topic at hand and the words that preceded it?
This general language capacity can then be honed through more specific datasets. Ask ChatGPT to consume educational research in the What Works Clearinghouse, for example, and the chatbot can become an expert pedagogical advice-giver. The better the data, the better the output.
At the fringes of this conversation, we approach what once seemed like science fiction. Some schools in China already require students to wear biofeedback headbands that send information to teachers about who is paying attention, who is angry, who is daydreaming, or who is drooling on their desk. Pair that with the recent advances in brain imaging that have produced relatively accurate text from brainwaves alone, and soon our own private thoughts will not be so private.
There are obvious concerns about data breaches, as when hackers accessed an online test-proctoring platform and subsequently leaked private information about 444,000 students. Similarly, glitches in a popular program that lets teachers view and control student screens would allow hackers to gain access to students’ webcams and microphones.
But I’m more concerned about a more abstract threat: How does constant surveillance affect the way we think, speak, and behave? French philosopher Michel Foucault popularized the concept of the Panopticon, a theoretical, circular prison with a watch tower in the middle that shines bright lights into exposed cells. Prisoners feel a constant sense of surveillance and so act as if they’re always being watched—fostering a compelled self-regulation into “proper” behavior.
What happens when our students really are being constantly monitored? A report from the National Association of State Boards of Education suggests that students are less likely to feel safe enough for free expression, and that these security measures “interfere with the trust and cooperation” and cast “schools in a negative light in students’ eyes.”
What district policies do to address student privacy concerns do not even begin to reckon with such threats. Gesturing at FERPA and warning students not to enter private information into LLMs, as Oregon has done, is insufficient, but at least the state has begun thinking through policies and recommendations. The next phase should be to develop sensible systems and regulatory frameworks that could handle these kinds of difficulties.
Software companies can create the skeletons of the tools to sell to districts, which already maintain private records. Consent forms could allow access to plenty more data, but such forms should be clear and understandable, not the inscrutable tomes that everyone currently ignores. Regulation must compel companies to collect only necessary data, limit their ability to sell this information, and dictate not just parameters for data collection and retention, but also deletion. There must also be aggressive legal consequences for firms that mishandle such data and suffer breaches.
But questions still remain: Will data stores like those held by the National Center for Education Statistics become public utilities? Will we need “freedom of thought” court rulings and regulatory frameworks in the coming years? Will student “credit scores”—currently created and tracked by some districts using AI software—get passed along to police when they graduate?
Asking these questions makes me feel like I’m a bit of a crazy person. But these are not issues to face in some distant future of some science fiction novel. Many are here now, and waving away generative AI as just another forgettable edu-tech promise risks exposing students to all of the hazards of AI without harnessing its potential benefits.
