Can AI Really Get the Jokes? How Idioms Affect Essay Scoring in Generative AI
Introduction: The New Wave of AI in Education
Artificial Intelligence (AI) has dramatically reshaped how we interact with technology, and it's making waves in the education sector as well. From helping students with personalized learning plans to automating boring administrative tasks, the possibilities seem endless. But what about using AI to score essays?
That’s the big question addressed in a recent study by Enis Oğuz, “Can generative AI figure out figurative language? The influence of idioms on essay scoring by ChatGPT, Gemini, and Deepseek.” It turns out that while generative AI technologies like ChatGPT and their newer cousins can do quite a lot, they may struggle with the tricky world of figurative language, especially idioms. So, let’s dive into what this research unpacked and what it means for students and educators alike!
What’s the Big Deal About Idioms?
Idioms are phrases that don’t make sense when taken literally. Ever heard the phrase “kick the bucket”? Well, it doesn’t mean someone is going to play a game of soccer with a bucket! It’s a colorful way to say someone has passed away. These expressions add depth and emotion to language, but they can be a nightmare for anyone—or anything—trying to interpret communication accurately.
The challenge for generative AI models, like ChatGPT and Gemini, lies in understanding these idiomatic expressions. If they misinterpret or overlook figurative language in essays, it could negatively skew their scoring, potentially penalizing students who skillfully use these expressions.
The Study: A Look Under the Hood
Setting the Stage
Oğuz's study involved this glaring inquiry: How does the presence of idioms impact the reliability of scores given by generative AI models? To tackle this, the researchers analyzed 348 student essays—half contained idioms, while the other half didn't. They then asked three AI models—ChatGPT, Gemini, and Deepseek—to score these essays using the same rubric that human raters use.
Here’s What They Did
Essay Selection: They created two essay lists: one with idioms and one without. This ensured they could see how idioms influenced scores on an equal playing field.
AI Scoring: Each model scored every essay three times to check for consistency and reliability in scoring.
Human Scoring Comparison: They compared the AI scores with human raters to assess how much influence idioms had on the models’ scoring performance.
Key Findings: The Good, the Bad, and the Idiomatic
Consistency is Key
What did the data reveal? All three AI models tended to score essays lower than human raters, but here’s where it gets interesting: Gemini outperformed its competitors. The study found that Gemini had the best inter-rater reliability compared to human raters, especially for essays with idioms present. This indicates that it could understand idioms better than the others.
Why Gemini Shined Brightest
- Reliability: Gemini showed a significant level of consistency regardless of the idioms present. Scoring agreements were strong, reaching an ICC (Intraclass Correlation Coefficient) value of 0.735, which is quite impressive!
The Mixed Success of Other Models
ChatGPT and Deepseek, while still good, didn’t quite reach Gemini's level of understanding idiomatic expressions. These models struggled with essays containing multiple idioms, which seemed to diminish their reliability.
In simpler terms, when the essays got a bit "quirky" with idioms, ChatGPT and Deepseek got a bit lost, scoring essays less reliably than the human raters, and definitely less so than Gemini.
Idioms Matter, but Not Always the Same Way
An interesting finding from the study was the "penalty" effect of idiom overuse. Initially, essays with idioms boosted scores. However, once the idioms became repetitive, the scores took a nosedive—not just for AI models but also for human raters.
Visualization of Scoring Patterns
The graphical representation of the scoring patterns showed an intriguing trend: human scores initially rose with the use of idioms but dropped dramatically as repetitive idioms increased. Gemini surprisingly tracked this trend closely until it outperformed humans at extreme levels of idiom repetition.
Real-World Implications: What This Means for Students and Educators
So, why should we care? The implications of OÄźuz's research create significant conversation points for educators and students alike:
1. Importance of Training in Figurative Language
If you're a student who loves to use idioms (and let’s be honest, who doesn’t want to sound more poetic?), understanding how AI interprets these expressions is crucial. Using idioms responsibly can actually enhance your writing and score, but overusing them may backfire.
2. AI as a Flexible Scoring Tool
Educators looking to implement AI for scoring essays should keep an eye on which models they use. Gemini appears to be the most promising candidate for scoring essays that involve rich figurative language. It could help save educators time while still accounting for the nuances of writing styles.
3. Room for Improvement in AI Technology
For developers, this study sheds light on the need for enhancing AI models to grasp figurative language better. Understanding idioms could truly enhance learning experiences, making AI not just a tool but a fantastic collaborator in education.
Key Takeaways: What We Learned
Understanding Idioms: The research reveals that while generative AI can score essays, its understanding of figurative language, specifically idioms, is inconsistent.
Gemini as a Star Performer: Among the three AI models studied, Gemini proved to be the most reliable for scoring essays containing idioms, closely mirroring human raters.
The Double-Edged Sword of Idioms: Using idioms can boost writing scores, but overusing them can diminish essay quality, affecting evaluations from both humans and AI.
Future of AI in Education: This study highlights the potential for hybrid approaches in scoring essays that combine AI's efficiency with human raters' nuanced understanding, particularly in recognizing figurative language.
So next time you weave a clever idiom into your writing, remember—the AI might be just a bit confused, but with evolving technologies like Gemini, the gap is starting to close!
And hey, whether you're a student or an educator, understanding how to work with AI will certainly help you leverage this technology to your advantage.
