Generative AI vs Real Intro OOP Exams: What Improved (2026)

Generative AI vs Real Intro OOP Exams: What Improved (2026)

Introduction: GenAI can pass Java OOP—but not the hard parts

If you’ve been wondering whether generative AI (GenAI) is “actually good” at intro Object-Oriented Programming (OOP), this new 2026 research is a reality check. The study behind it, from arXiv:2608.16318, tests several popular large language models on authentic university OOP assessments—not made-up benchmarks—and compares the results to real student performance from the same course.

What makes this paper especially useful is that it doesn’t just ask, “Did the model generate code?” It grades AI outputs using the same rubric used for students, including the rule that non-compiling code earns zero points. Even better: the researchers run a longitudinal comparison against an earlier evaluation from the previous year, using the same assessment tasks.

The headline? In general, five widely used GenAI systems scored higher than the historical student average on these introductory OOP tests—often hitting full marks on longer programming tasks. But the story gets more interesting (and more nuanced) when you look at what still trips the models up: interfaces, abstract classes, and any OOP question tied to interpreting visuals.

This post breaks down what the researchers did, what they found, and what it means for teachers, students, and anyone designing programming assessments—based on the study in the original paper.

Why This Matters: The “AI can code” story is becoming less useful than “AI can reason”

This research is significant right now because we’re past the stage where the big question is whether AI can write Java. Most models can. The new urgent question is: Can they meet the hidden constraints that exams and rubrics care about?

In practice, that’s what separates “chatty code” from real competence. Intro OOP assessments force students to show understanding of abstraction and program structure through things like interface vs abstract class, correct inheritance rules, and (in exam form) correct interpretations of what code will do. Models may generate syntactically plausible solutions, but the paper shows they still sometimes miss subtle requirements—especially when multiple answers are possible and the exam expects you to mention them all.

A very real scenario where this applies today: think of a university course that wants to keep assessments meaningful while allowing student tool use. If instructors switch to tasks that only reward “working code,” GenAI will likely do better than students and reduce the value of the grade. But if instructors instead design assessments that test conceptual constraints—where answers require careful reasoning about OOP rules and correctness under strict rubric checks—this paper suggests GenAI will still struggle more than you might expect.

Finally, this builds directly on earlier research showing inconsistent reliability as tasks become harder. The paper explicitly compares results against the previous year’s evaluation and finds substantial improvements across most systems, meaning conclusions from last year can go stale fast. That longitudinal angle is a big deal for anyone trying to decide what “good” looks like in 2026, not 2025.

What the researchers actually measured: five GenAI systems, real course tasks, strict grading

The study evaluates five GenAI systems on an Estonian-taught, Java-based intro OOP course at the University of Tartu:

• ChatGPT-5.2
• DeepSeek-V3
• Gemini 2.5 Flash
• Claude Sonnet 4.5
• M365 Copilot (paid, but available for free to students via university accounts)

The course assessments: tests vs final exam

The course includes programming tests and a final exam. This study focuses on the programming tests and the final examination, because they are more standardized.

Programming tests
- Two tests during the 16-week course (weeks 7 and 13)
- Students typically have 105 minutes
- Tests include a program outline that specifies required classes/methods and the overall structure
- Students can use course materials but no AI tools
- Crucially: if the generated program does not compile, it gets zero points (same for AI)

Final exam
- Computer-based exam on Moodle
- 60 minutes
- Students may use course materials and Google search, but not AI assistants, and not open IDEs
- Includes:
- randomized short questions (multiple-choice, fill-in, matching, etc.)
- a longer open-ended end task worth 6 points

The input style: no prompt engineering, full original text in Estonian

A key design choice: the researchers submitted the full text of the original tests and exam questions to each chatbot with no modifications, no translation, and no prompt engineering. They also evaluated only the first response each system produced, reflecting realistic “paste the question and use what you get” usage.

They then graded AI outputs using the exact same grading scheme the instructor used for student submissions. Common errors were also recorded to identify recurring limitations.

Sample sizes and comparison baseline

• For Programming Test 1, the study notes 285 students completed it.
• The researchers compare AI scores to:
  1. the historical student cohort from the same course
  2. and a previous year evaluation of ChatGPT and M365 Copilot

That comparison matters because it helps you interpret whether AI is merely “better than average” or meaningfully stronger than what students typically achieve under the course’s own expectations.

Programming Test 1: Most models clear the bar—and Copilot still can’t always compile

Programming Test 1 is worth 16 points, with 12 required to pass. It covers core Java OOP topics including classes/objects, strings, file handling, lists, polymorphism, interfaces, and abstract/super/subclasses.

Overall performance (and the compilation landmine)

On average, nearly all systems did fine. The main exception: M365 Copilot, whose average was dragged down by one non-compiling solution that earned 0 points.

Here are the reported averages (with the historical student average shown for context):

What went wrong (repeatable issues show up)

Even when scores are high, the paper records recurring failure modes:

• Encoding mistake (repeat across models):
  ChatGPT and DeepSeek both failed to specify the required file encoding. That’s a classic “implementation constraint” problem: the code can look right but still violate a requirement the rubric expects.
• An improvement vs the previous year:
  For ChatGPT, a prior-year recurring issue was fixed. Previously it incorrectly placed logic in abstract methods of the superclass instead of leaving them to subclasses (the paper says this error “was avoided” this year).
• Copilot compilation failure details:
  The non-compiling error was a toString syntax issue involving string concatenation with three consecutive single quotes. The paper suggests that once corrected, Copilot would likely have scored 15 points, but as graded, it hit 0 for compilation.

The subtle behavioral differences: “over-helping” code

Not all observations were treated as errors, but they matter for assessing tool behavior:
- Several models generated getter/setter methods even though the task specified creating them only if necessary.
- Some generated extra methods not explicitly requested.
- One interesting inconsistency: streams were not taught before Test 1, and students were expected to use Scanner. The paper says only Gemini used Scanner; the others used BufferedReader and FileReader.

Bottom line for Test 1: GenAI can reliably produce working OOP programs in a rubric context—but you still see consistent slips around constraints like encoding and around strict “must compile” rules.

Programming Test 2: Higher complexity, still mostly strong—except Claude hits a compile issue

Programming Test 2 expands the topics: streams, exception handling, and data structures, plus continued coverage of earlier material (Test 1 topics). It’s worth 16 points and, unlike Test 1, has no minimum pass score.

Average results: near-full scores dominate

Most models are again very high. Claude’s average drops due to one non-compiling solution that received 0.

Recurring mistake: visibility keywords (public vs private)

A repeated issue shows up here again:

• ChatGPT marked methods as public when the task required private. The paper notes the requirement was explicitly stated in the test.
• DeepSeek had the same visibility issue in one solution.

Implementation details: models use “beyond course” patterns

The paper points out that many models used programming constructs students likely weren’t taught yet:
- sorting via Comparator.comparingInt and method references :: (not core course materials)
- computing averages and grouping with stream pipelines like list.stream().mapToInt(...).average() and Collectors.groupingBy(... )
- using methods like computeIfAbsent, putIfAbsent, and removeIf

This matters practically: students could be learning a specific “expected path,” while models choose flexible alternative implementations that still satisfy the rubric.

Final exam results: GenAI shines on basics, but struggles on interfaces/abstract classes and graphics

The final exam covers the full course, mixing short conceptual questions and a longer open-ended reasoning task. The paper reports the average points for each model.

Exam averages: improvement, but no one reaches the top tier

This year’s exam results show all models improving compared to the previous evaluation year for ChatGPT and Copilot, but still not dominating:

Yet the paper also notes a ceiling:
- None of the AI tools reached the top 10th percentile.

So yes, they often do better than students overall—but they don’t “max out” at the very highest student performance level.

Where models consistently lose points: OOP reasoning (interfaces/abstract classes) and visual interpretation

The exam breakdown highlights a pattern: questions about interfaces, abstract classes, and graphics remain hardest.

Interfaces and abstract classes: recurring conceptual misconceptions

In multiple exam question sets (labeled in the appendix as topics Q4, Q5, Q8, Q9), models repeatedly make related mistakes, such as:

• thinking a class must implement all interface methods just because it has a method with the same name
• assuming an abstract class must implement all interface methods
• confusion about extends vs implements
• claiming interfaces cannot contain constants (the paper says all assistants got that wrong)
• insisting abstract cannot appear “within an interface” (again, recurring across most assistants)

This is a big deal educationally: those are exactly the “threshold concept” OOP ideas students struggle with too.

Graphics questions: multimodal reasoning is still shaky

For graphics-related questions (Q10), the exam used images produced by a JavaFX program—passed to the systems as a single screenshot.

The paper’s result: all models struggled, and these were the only questions that got zero points by at least one system. It also describes concrete failure patterns:
- misreading the position of shapes (black/red swapped)
- failing to choose the correct output image based on a code snippet
- sometimes able to describe expected output in words, but not select or generate the correct visual outcome

The key practical takeaway: even if a model can explain what should happen, visual interpretation tasks remain a weak spot.

Common error patterns: what the AI keeps getting wrong (and what students also struggle with)

Across programming tests and the exam, the paper groups recurring mistakes into several buckets:

1. Interfaces and abstract classes

   • misconceived requirements for implementation
   • misuse of extends vs implements
   • misunderstandings around @Override and the meaning of abstract

2. Incomplete interpretation of task constraints

   • generating only one valid answer when the exam expects all valid options
   • missing “quiet requirements” like file encoding or visibility rules
   • failing to mention multiple correct access modifiers in the long-form reasoning question

3. Occasional implementation/compilation mistakes

   • missing imports (e.g., Claude in Test 2)
   • syntax issues causing non-compiling programs
   • sometimes generating extra unused methods/variables

4. Visual reasoning

   • graphics/image interpretation errors on Q10

Interestingly, the paper points out that these recurring AI misconceptions resemble well-documented novice programming misconceptions, suggesting a potential shared gap: GenAI can imitate solutions, but it may still “pattern match” the wrong mental model when the concept is genuinely tricky.

Key Takeaways

• GenAI is now reliably strong on many authentic intro OOP programming tasks. In Test 1 and Test 2, most models scored higher than the historical student cohort.
• Compilation and rubric constraints still matter. M365 Copilot lost a lot of points due to non-compiling output in both Test 1 (one 0-point case) and Test 2 wasn’t affected the same way—but compilation risk remains real.
• The consistent weak spots are conceptual OOP topics: interfaces and abstract classes keep producing recurring errors across multiple systems.
• Graphics/visual interpretation is still unreliable. The exam’s image-based questions (Q10) were the only ones where systems could score zero, and models often misread screenshots even when they can describe behavior in words.
• Long-form “all valid answers” reasoning trips models up. In the final reasoning question (Q15), systems often mention only one correct path (like missing some access modifiers or failing to discuss both superclass and subclass typing options).
• For instructors and assessment designers: if you want future-proof evaluations, don’t only test “can the model produce code.” Test the conceptual constraints where the models still fail—especially around interface/abstract semantics and multiple-correct-answer reasoning.
• For students using AI: treat GenAI output as a draft. Verify interface/abstract-class logic, access modifiers, and any requirement that the rubric might enforce implicitly (like file encoding or “must mention all valid options”).

If you want, I can also turn this into a practical checklist you can use to audit AI-generated Java submissions for the exact failure modes this paper observed (interfaces, extends vs implements, abstract method rules, and graphics interpretation traps).