Can an AI reliably detect whether a student is frustrated, motivated, or indifferent? This question was at the heart of our empirical study, presented at the EDULEARN25 conference in Palma de Mallorca in July 2025.
Sentiment analysis as a core component of SMARTA
In the SMARTA project, we develop AI-powered chatbots that support students with motivation, learning organization, and personal challenges. For the chatbots to respond empathetically to procrastination, social pressure, or frustration, they need to accurately detect the emotional tone of conversations. That is where automated sentiment analysis comes in.
What the study examined
In a comparative study, sentiment classifications by linguistic experts, students, and various GPT models (including GPT-3.5 and newer versions) were analysed. Quality metrics included accuracy, precision, recall, and confusion matrices.
Result: GPT more reliable than expected
Even older models like GPT-3.5 recognise sentiment with high reliability. Deviations occur primarily in neutral statements — an area where human assessments also diverge. This led us to propose the Human-AI-Gap-Benchmark: AI performance should be measured relative to human error rates, not against a zero-error ideal.
Legal context: EU AI Act
Sentiment analysis remains a black-box process with limited explainability. We assessed the method for compliance with the EU AI Act. Our solution: sentiment results are stored anonymously, and the chat history is deleted immediately after each conversation ends. In the SMARTA project, sentiment analysis is deployed as a trusted module.