School software surveils students in and out of school
Minneapolis Public Schools are using software to monitor student communications in and out of school, raising serious concerns over student privacy, according to the non-profit The 74 ,that has analyzed public records and just issued a report that raises serious concerns related to the use of the Gaggle software that can be used for 24-hour monitoring through school-provided tech devices, and includes Identifying and passing along student interest in keywords including “gay”, “LGBTQ” and others.
“It suggests that anything you write or say or do in school — or out of school — may be found and held against you and used in ways that you had not envisioned,” Jennifer Mathis, director of policy and legal advocacy at The Bazelon Center for Mental Health Law, told The 74.
Gaggle uses “artificial intelligence' techniques to analyze students' emails, chat messages, documents, personal files and media looking for words and terms that its dictionary of terms indicates they are at risk or potentially broaching a concerning topic. If a term is identified, a human moderator is notified and evaluates the flagged content before deciding to pass it along to school officials or not.
This process is endorsed by Gaggle CEO and founder Jeff Patterson as a way of helping endangered minors. “As a parent you have no idea what's going on in your head, but if you don't know you can't help them, and I would always want to err on trying to identify kids who need help,” Patterson said.
Gaggle's inclusion of LGBTQ terms in its library of questionable terms led to at least one LGBTQ student being outed to their parents by school officials.
Theory In Action
But the theory behind Gaggle's program may not exactly line up when put into action. An analysis of 1300 records from Minneapolis Public Schools showed that Gaggle system identified numerous uses of terms related to pornography, suicide and violence by students, including some from journal entries, classroom assignments and fictional stories. But Gaggle's software isn't able to filter those terms for context.
This resulted in school-assigned projects discussing mental health, short stories submitted to a school literary magazine and private discussions on cartoon porn between students being flagged and sent along to school officials. In the case of trans eighth-grade student Teeth Logsdon-Wallace, they told a reporter that this process re-traumatized him after overcoming severe depression and suicidal ideation triggered by remote learning last year.
In fact it was Logsdon-Wallace writing about overcoming those issues, his suicide attempt and the role music played in his recovery for a classroom assignment, that Gaggle's system flagged and notified school officials. When those officials contacted his mother, Logsdon-Wallace said he felt some betrayal.
“I was trying to be vulnerable with this teacher and be like, ‘Hey, here's a thing that is important to me because you asked,” Logsdon-Wallace told The Guardian. “Now, when I've made it clear that I'm a lot better, the school is contacting my counselor and is freaking out.”
Flagging “Gay” and “Lesbian”
The study by The 74 found that Gaggle's system is likely to disproportionately affect LGBTQ students. Three dozen records show that Gaggle flagged terms concerning LGBTQ identities, including the words “gay” and “lesbian.”
“When people are just talking about being gay, anything they're writing would be flagged,” Logsdon-Wallace explained. “They have ‘gay' flagged to stop people from looking at porn, but one, that is going to be mostly targeting people who are looking for gay porn and two, it's going to be a false positive because they are acting as if the word gay is inherently sexual.”
According to a Minneapolis school newspaper, Gaggle's inclusion of LGBTQ terms in its library of questionable terms led to at least one instance where an LGBTQ student was outed to their parents by school officials.
They have ‘gay' flagged to stop people from looking at porn, but that is targeting people who are looking for gay porn and it's a false positive as if the word gay is inherently sexual.”
Teeth Logsdon-Wallace
Gaggle maintains that its service saves lives, though its claim that Gaggle did so for 1,400 minors during the 2020-21 school year have not undergone any form of independent verification. The company experienced growth of more than 20% in 2020 after a number of school systems implemented the program during remote learning.
The 74's study found that roughly a quarter of flagged incidents reported to school officials happened when students would be on school grounds, but in spite of the fact that a majority of reports were generated at home, Minneapolis school officials deny that students' privacy is undermined by its use of Gaggle tied to school-issued accounts and devices.
That “privacy tax” on those who economically are more reliant on a school supplied device means that low-income students are more likely to be surveilled along with the LGBTQ ones, simply because a school-supplied device is more likely to be their only device for personal as well as academic work. This in turn increases their likelihood of interfacing with the justice system which has been shown to increase the likelihood of being further disproportionately surveilled through adult hood
“That's definitely really messed up, especially when the school is like, ‘Oh, no, no,no, please keep these Chromebooks over the summer,” Logsdon-Wallace Said.
Gaggle: Previously on Towleroad
Photo courtesy of Fibonacci Blue/Creative Commons
