Detecting AI
...
Created Content
Where we stand — February 2024
ITS and Online Learning Services are acutely aware that artificial intelligence tools like ChatGPT have been disruptive to the assessment methods used by many instructors. We have long been aware of the risk of ghostwriters and creative solutions for cheating on exams, but the ease with which these new AI tools can generate believable content has led to a sharp increase in questions about how to determine that the work submitted by students is original (Mills 2023). The detection of AI content is notoriously difficult(Edwards 2023; Heikkilä 2022).
...
In the context of academic integrity, the risks of false positives are significant (Klee 2023; Fowler 2023). Unreliable AI detection not only fails to improve academic integrity but may deepen existing inequalities. Non-native English speakers are flagged by AI detection tools at a disproportionate rate(Myers 2023). Other tools like Grammarly with legitimate academic applications, particularly for writers with dyslexia and other learning disabilities, also increase the likelihood of being flagged by AI detectors (Shapiro 2018; Steere 2023).
What to do?
All this leaves instructors in a challenging position where the best recommendations being put forward are to redesign their assessments. Redesigning assessments is difficult and time consuming, and the new assessment methods often require more time to grade. Just as AI tools are beginning to make the process of writing faster and easier for everybody, it feels unfair that teachers of writing are forced to spend more of their own precious time on addressing the downsides and potential misuse of these tools.
...
Online Learning Services will continue to evaluate new teaching and learning technologies and remains available to consult with faculty on teaching and technology. ITS will continue to provide access to effective tools where they are available. In addition to technological considerations, the Center for Teaching and Learning Excellencehas pedagogical and policy resources for instructors on strategies they might take to improve their assessments (CTLE, n.d.).
AI created content detection and Turnitin
In early April 2023, Turnitin released an AI writing detector. This tool was enabled in the Syracuse University Turnitin system without our consent and without the ability to disable it. There as a preview. During the preview there were no fees initially associated with the tool. Turnitin initially reported low rates of false positives, but those have since been called into question. (Chechitelli 2023; D’Agostino 2023). The detector’s false negative rate was close to 40-50% in tests where AI-generated text was reworded by a human or by a separate AI paraphrasing tool(Weber-Wulff et al. 2023).
In late At the end of the free preview on December 31, 2023, Turnitin announced that it would begin charging an additional license fee for the use of the AI Detection tool as of January 2024. Given the concerns about its effectiveness, ITS elected to forgo the considerable additional fees associated with re-acquiring the not license the AI content detection tool. We are not alone in this choice as multiple R1 universities have made a similar decision (Brown 2023; Coley 2023; “Known Issue – Turnitin AI Writing Detection Unavailable – Center for Instructional Technology | The University of Alabama” 2023).
At present ITS has no intention of acquiring a license to enable Turnitin’s AI Detector. We are also unable to recommend any alternative technological solution. None of the AI detection tools currently available online are accurate enough to provide credible evidence in academic integrity investigations. The risk of misleading results harming students who are acting in good faith is too great. ITS is committed to thorough and transparent vetting of any new tools that emerge in the future. If and when a reliable tool for AI detection becomes available, ITS will present it to evaluate the campus tool and carefully consider recommending it to the Syracuse University academic community.
Bibliography
“Authentic Assessment.” n.d. Center for Innovative Teaching and Learning. Accessed February 27, 2024. https://citl.indiana.edu/teaching-resources/assessing-student-learning/authentic-assessment/index.html.
...