By Ian Linkletter, emerging technology and open education librarian, BCIT
On September 18, 2024, as part of the BCcampus EdTech Sandbox Series, I presented my case against AI proctoring and AI detection. In this post you will learn about key points from my presentation and our discussion.
As I disclosed at the beginning of the session, the AI proctoring company Proctorio filed a lawsuit against me in 2020 for criticizing them using links to publicly-accessible YouTube videos. It continues to this day.
The case against AI proctoring
Moving beyond surveillance requires us to be aware of its past harms. Sometimes, a technology is so abhorrent that there is no ethical choice but to discontinue its use. AI proctoring is one of those technologies.
During the pandemic, some institutions turned to AI proctoring to surveil students in their homes as they took exams. AI monitors bodies, behaviour, and audio, scrutinizing head and eye movement for so-called “abnormalities”, which can harm students with disabilities. Each student is assigned a “suspicion level” based on proprietary behavioural flags such as changes in audio level or window resizing.
AI proctoring invades privacy, using facial detection to decide whether access should be granted. When widely implemented in 2020, many students of colour faced “face not found” errors that made their tests inaccessible.
Student Amaya Ross had to shine a flashlight in her eyes for Proctorio to recognize her. Amaya wasn’t alone. Robin Pocornie, Gavin Gordon, and Femi Yemi-Ese shared similar experiences with Proctorio. As Dr. Chris Gilliard writes, “Imagine all you want to do is take a test, and the system your institution uses as a gateway to testing doesn’t recognize you as a human being.”
The root of the issue is biased AI. Lucy Satheesan’s research showed Proctorio’s facial detection, based on an open-source computer vision software library, failed to detect Black faces 57 per cent of the time. Lucy’s methodology was validated by VICE Motherboard and RTL Nieuws, but refuted by the company.
In March 2021, UBC Vancouver Senate’s Teaching and Learning Committee found that “…facial detection algorithms can fail to detect students’ faces with darker skin tones preventing some Black students from accessing exams without extra intervention, thereby causing undue stress and harm. The committee has concluded that this is an unacceptable form of racial discrimination.”
Both UBC Senates voted to restrict Proctorio and other automated invigilation tools. Other schools have similarly abandoned invasive proctoring. Research has also shown that AI proctoring tools are not effective in detecting cheating. I believe this technology is unethical and has no place in education.
In our session, participants discussed:
- Can an algorithm be racist?
- How many “face not found” errors are acceptable?
- How are our institutions protecting students from discriminatory AI?
The case against AI detection
Plagiarism detection software like Turnitin has been around for over 25 years and, until recently, worked by scanning student work for similarities to other text. With the release of ChatGPT in 2022, it became possible to generate unique text that could not be detected by Turnitin’s existing product.
In 2023, Turnitin added an AI detection feature, which uses natural language processing to look for “perplexity”. This term is used to describe the unpredictability of human written text.
Turnitin’s own explanation of how its AI detection works is unacceptably vague: “…we give each sentence a score between 0 and 1 to determine whether it is written by a human or by AI. If our model determines that a sentence was not generated by AI, it will receive a score of 0. If it determines the entirety of the sentence was generated by AI it will receive a score of 1.”
False positives are a major problem when accusing students of plagiarism because of the harmful, or even tragic, impact false accusations can have on students. Since its launch, Turnitin’s AI detector has scanned over 200 million assignments. That’s over 8 million false positives, according to Turnitin’s own claim of a 4% rate of sentence-level false positives.
The true rate may be much higher. Turnitin’s AI detection was designed for GPT-3.5. Both the free and paid versions of ChatGPT now use GPT-4, which is a more advanced model. Turnitin has never disclosed its false positive rate for GPT-4 generated text.
The algorithms used by Turnitin and other AI detection software are not transparent, making it difficult to evaluate bias. Writing by non-native English speakers is more likely to be flagged as AI-generated. Turnitin refutes this research, which was not conducted using their product, but has not made its source code or product available to academic researchers.
Students are aware of the limitations of AI detection. They are using paid Large Language Models (such as ChatGPT Plus) and “word spinner”, “humanizer”, or “rewriter” tools to circumvent it. This exacerbates existing equity issues because students with money and digital literacy can gain an advantage.
In British Columbia, UBC and BCIT have both rejected the use of AI detection.
Participants discussed:
- How can we build a culture of academic integrity that is built on trust, not suspicion?
- How might AI surveillance technologies undermine mutual trust between teachers and students?
I believe that surveillance does not have a place in education. As individuals, we can move forward beyond surveillance through the power of collective refusal.