AI for Coding: Better Grades, Worse Memory, Less Ownership

Generative AI is showing up everywhere in student work—especially programming. New research from the arXiv paper “Your Programming Students' Cognition with ChatGPT” (Christian Bergh, Benjamin Tag, Alexandra Vassar, Jake Renzella) digs into a question that many educators and students feel but haven’t always been able to prove: does AI help students perform better without helping them actually learn?

In a controlled experiment with 59 undergraduate computer science students (and 55 retained for analysis), researchers let some students use ChatGPT-4.5 while others used conventional web search (with AI features disabled). The twist is that the team didn’t just grade the code—they also tested retention immediately and 48 hours later, measured cognitive load using both self-reports and physiology, and asked students how much they felt the submitted work was their own.

What they found is a big deal for programming courses: students using ChatGPT got higher coding scores (89% vs. 69%)—but they remembered less (41% vs. 53% immediately; 39% vs. 52% after 48 hours) and felt less ownership of their code (self-attributed ownership: 45% vs. 81%). Even when the “work looked right,” the learning signals weren’t matching the performance signals.

Why This Research Matters (Right Now, In Real Courses)

This research is significant now because programming education is shifting fast from “practice to learn” toward “produce to prove.” Many courses still treat summative assessment (grades on assignments/exams) as a clean proxy for learning. But if AI can boost output without boosting durable understanding, grades can become misleading—almost like measuring typing speed when what you care about is reading comprehension.

A concrete scenario: imagine a first-year programming course where students can use AI during homework. After a few weeks, the instructor notices—“Great! The class average is higher. Fewer students are failing. Everyone is coding correctly.” But then the next day (or two days later), you run a quiz that asks students to explain edge cases, recall key logic, or modify a solution without AI help. Suddenly performance drops. This paper gives an experimental blueprint for why that happens: AI-assisted students can complete tasks successfully while storing less of the task-specific knowledge.

It also builds on earlier AI education research that often shows performance boosts and mixed learning results, but this study adds a missing layer: ownership and cognitive engagement. Prior work commonly focuses on outcomes like accuracy and perceived difficulty. Here, researchers measured what students retain and whether they psychologically “own” what they produced—finding a consistent disconnect.

In other words, this isn’t just “AI helps and may reduce learning.” It’s “AI changes the learner’s relationship to the task,” which is exactly what educators need to understand if they’re designing assessments for an AI-present reality. (The original paper is worth reading for the full experimental detail.)

What the Experiment Compared: ChatGPT Assistance vs. Conventional Search

The researchers ran a between-subjects experiment. Students completed three introductory C programming tasks meant to resemble homework work.

Here’s the core comparison:

A few key design points matter because they make the comparison cleaner:

• The workspace was split: one side for the programming environment (EdStem) and one side for the assistance tool (ChatGPT or Google).
• Students were told they could copy/paste instructions and scaffold code into the assistance panel.
• The AI group used ChatGPT-4.5 with memory off and fixed model settings.
• The no-AI group could use Google search, but major GenAI components were disabled in-browser.

Sample and retention for analysis
- Total recruited: 59
- Retained for analysis: 55
- AI-Assisted: 29
- No AI: 26

What Students Did Better: Coding Scores Jumped with ChatGPT

On the actual programming assessment, ChatGPT-assisted students performed substantially better.

• Overall coding score: 89% (AI-Assisted) vs. 69% (No AI)
• This difference was statistically significant (Welch’s t-test, p < .001, with Holm correction still significant).
• Effect size was large (Hedges’ g = -1.24).

This supports a pattern many educators already see: with LLM help, students can more easily produce correct or near-correct solutions—even when tasks are non-trivial.

But here’s where the story stops matching the usual “more practice = more learning” assumption.

Some students in the AI condition didn’t always finish tasks because they used ChatGPT in a “tutor-like” way—asking it for questions or hints rather than solutions. In one case, the interaction became circular and self-contradicting, preventing the student from completing later tasks. In the control group, a smaller number attempted to use Google as if it were an AI summarizer—without that AI feature available, the result wasn’t helpful and also prevented completion of the hardest task for a couple participants.

Still, at the group level, AI increased task success.

The Big Catch: Retention Was Lower Immediately and After 48 Hours

Performance and retention are not the same thing. The researchers tested content retention using cued recall quizzes immediately after submission and again 48 hours later.

Immediate retention (right after the task)

• No AI recalled 53%
• AI-Assisted recalled 41%
• Difference significant (Welch’s t-test p = .010, Holm-adjusted p = .011)

Delayed retention (48 hours later)

• No AI recalled 52%
• AI-Assisted recalled 39%
• Difference again significant (Welch’s t-test p = .005, Holm-adjusted p = .011)

And crucially: the decline from immediate to delayed recall looked similar between groups. In the paper’s mixed analysis, there was no significant group × time interaction—meaning both groups forgot at roughly the same rate, but the AI group started lower.

So it wasn’t that the AI group “forgot faster.” It’s that they retained less from the start.

A vivid example: recalling their own code

One recall question asked students to recall a line of code from their submission.

• Immediately after submission:
  • 85% of No AI students could accurately recall a line
  • 48% of AI-Assisted students could do the same
• After 48 hours:
  • Accurate recall was 65% for No AI
  • 45% for AI-Assisted

Again, the AI group wasn’t just slightly worse. In several measures, the gap is large enough to matter for assessment design.

Cognitive Load and Effort: Less Mental Climb with ChatGPT

To figure out why retention might drop, the study measured cognitive load in two ways:

1. Self-reported mental effort and difficulty after each task.
2. Physiological signals during the session:
   • pupillary response (eye tracking)
   • heart rate variability (HRV)

Self-reported effort tracked the retention gap

The researchers found a significant change in self-reported mental effort across tasks for the AI group. Specifically:

• Mental effort increased across tasks for both groups—but the AI group’s increase was smaller.
• The key finding: a significant tool × task interaction for self-reported mental effort (chi-square p = .012, Holm-adjusted p = .047).

Follow-up analysis showed that as tasks got harder (Task 2 and Task 3), No AI students reported higher effort increases than AI-assisted students.

Difficulty ratings also rose with task complexity for both groups, but the effort pattern split.

Physiological measures were inconclusive (but informative)

The paper includes physiological measures as a check on whether self-reports match brain/body effort patterns. However, the physiological results did not show clear group-level differences. The authors flag this as likely data-quality-limited: many physiological intervals failed “quality gates,” and only a minority of measurements survived filtering.

So: physiology didn’t contradict the story, but it also didn’t provide strong confirmation. The researchers interpret physiological insignificance as inconclusive, not as evidence of no effect.

Ownership and Authorship: Students Felt Less “This Is Mine”

One of the most interesting parts of the study is that it measures psychological ownership—a student’s felt sense that the work they submitted is an extension of themselves.

In three post-task measures, ChatGPT use reduced ownership:

1) Self-attributed ownership (percentage of code they considered theirs)

• AI-Assisted: 45%
• No AI: 81%
• Difference significant (p < .001, Holm-adjusted p < .001)

2) Psychological Ownership Scale (POS)

• AI-Assisted: 2.92
• No AI: 4.30
• Difference significant (p < .001, Holm-adjusted p < .001)

3) Authorship attribution (who is the author?)

Students rated authorship from “sole participant” through AI as sole author.

Results were striking:
- In the No AI group:
- half claimed sole authorship
- others acknowledged Google, but students mostly stayed the primary author
- In the AI-Assisted group:
- only 7% claimed sole authorship
- 38% attributed primary authorship to ChatGPT or co-authorship with it

This matters because ownership isn’t just a feelings metric. In learning science terms, it likely tracks whether students engaged in the kind of cognitive work that leads to durable memory. If you don’t feel like you built it, you probably didn’t mentally construct the underlying schemas as strongly.

The paper ties this together: lower effort increase + lower retention + lower ownership all point to an “offloading loop,” where students can complete tasks while shifting the heavy lifting to the tool.

What This Means for Teaching: Designing Assessments That Measure Learning, Not Just Output

If you only look at grades, ChatGPT-assisted students in this study did great. But if you look at what remains after 48 hours—and whether students feel responsible for the work—the story flips.

So what should educators do? The paper suggests practical design directions, and the findings support them strongly.

1) Don’t treat summative output as a perfect proxy

This is the core implication: output quality ≠ learning quality, especially when AI is available. If your course heavily relies on assignments that students can outsource, you may unintentionally reward the ability to use tools—not the ability to understand.

2) Add assessment moments that require retrieval and explanation

Because recall was lower immediately and after 48 hours, courses need more tasks that force students to:
- retrieve the logic without the tool
- explain edge cases
- justify modifications

Even short “closed-book” checkpoints can help, because this study shows the retention gap exists even after controlling the forgetting window.

3) Build in critique-and-verification steps

The authors recommend designs where students must actively engage with AI output—testing, validating, and justifying changes—rather than pasting solutions.

A “critique-the-AI” phase could look like:
- “Run this code on edge cases we specify.”
- “Explain why this function behaves correctly.”
- “Modify the function to handle an input range and predict what changes.”

This shifts cognitive activity back toward schema-building and ownership.

4) Protect student autonomy while preventing cognitive offloading

It’s not enough to say “use AI less.” Students will use it. The goal is to scaffold cognitive development rather than replace it.

A practical classroom stance could be:
- AI is allowed for drafting, but students must complete a human-interpretation step before submission.
- Students must mark what they changed, why they changed it, and what they learned from the process.

Key Takeaways

• AI boosts coding performance: ChatGPT-4.5 users scored 89% vs. 69% on the C programming assessment (55 analyzed participants).
• AI reduces retention: recall dropped for the AI group both immediately (41% vs. 53%) and after 48 hours (39% vs. 52%).
• Ownership drops too: students felt less responsible for the submitted code (45% self-attributed ownership vs. 81% without AI).
• Cognitive offloading is likely: AI-assisted students reported a flatter increase in mental effort across harder tasks, consistent with delegating cognitive work to the tool.
• Grades can mislead: relying on assignment output as a learning proxy becomes riskier in AI-enabled environments.
• Assessment should measure understanding and retrieval: add “no-AI” explanation/checkpoint moments, and require critique/validation of AI-generated code.
• Future learning systems should scaffold—not replace—thinking: designs that force active contribution can help preserve durable learning and ownership.

If you’re designing a programming course (or even just running a study session for students), this paper is a strong warning label: correct code isn’t the same thing as learned code—and students will notice the difference when asked to remember.