Secure Coding with ChatGPT: Better Grades via Diverse Use

Secure Coding with ChatGPT: Better Grades via Diverse Use

Introduction: What “good ChatGPT use” looks like in secure coding

If you teach—or study—secure software engineering, you’ve probably had the same worry: students can get code that “works,” but still introduces risk. That tension is exactly why the adoption of Large Language Models (LLMs) is so intense right now. Tools like ChatGPT can generate code, explain security concepts, help debug errors, and offer quick feedback. But in security tasks (like vulnerability fixing), a technically correct patch can still be wrong in a deeper way—either leaving the vulnerability open or creating a new weakness.

New research from the paper posted on arXiv takes a closer look at this problem. Instead of asking a simple question like “Do students use ChatGPT?”, the study digs into how students used ChatGPT while fixing vulnerabilities in an assignment. That distinction matters, because “using ChatGPT” can mean anything from thoughtful cross-checking to copying-and-moving-on.

The study is exploratory, but it’s also surprisingly practical: it examines 26 graduate students in a part-time MSc Cybersecurity programme who used ChatGPT during a vulnerability-fixing exercise. The researchers analyzed students’ ChatGPT interaction logs, categorized different usage strategies, and then asked whether those strategies—and prior cybersecurity experience—had any relationship with assignment performance. The headline result is a strong hint that diversity of usage patterns may matter more than any single “best” pattern.

Why This Matters: The real issue isn’t access—it’s interaction strategy

This research is significant right now because many courses are still trying to solve the wrong problem. The common policy debate is binary: allow ChatGPT or ban it. But this study points to something more nuanced: what students do inside the conversation may be what determines whether LLM help turns into real learning and solid security outcomes.

Here’s a scenario you could apply today. Imagine a secure coding module where students must fix vulnerabilities in a web app using an instructor rubric and an OWASP-based set of issues (the assignment in this study was based on OWASP Top 10 2021 and relevant CWE entries). If students are only trained to ask ChatGPT for “the fix,” you may get plausible-looking patches—with limited security reasoning. But if you structure assignments so students must use ChatGPT in a multi-step, task-oriented way (e.g., ask for the relevant mitigation, request an explanation, refine based on errors, check whether the patch truly addresses the vulnerability), you’re more likely to steer them toward reflective problem-solving. The study’s results support that direction: students who used more distinct usage patterns tended to score higher.

This also builds on earlier AI education research that didn’t find that “having access to AI” automatically improves learning. It’s moving the conversation forward—from tool availability to interaction quality. And that’s a big deal for security education, where correctness isn’t just passing tests; it’s preventing the vulnerability in the way the rubric (and reality) expects.

How the study measured ChatGPT behavior during vulnerability fixing

The researchers didn’t treat ChatGPT usage as a yes/no variable. Instead, they took students’ ChatGPT conversation histories and converted them into a set of coded usage patterns. Think of it like this: rather than labeling a driver as “used GPS,” they looked at whether the driver used GPS for route planning, recalculation after a wrong turn, checking traffic updates, or just copying a destination address and hoping for the best.

Participants and the task setting

• Sample size: 26 graduate students
• Programme context: part-time MSc Cybersecurity module
• Prior expertise measurement: a pre-test questionnaire with responses to 9 cybersecurity questions
• Expertise grouping: students were classified into two groups:
  • medium-low expertise: scores in [0, 3] (34.6% of participants)
  • high expertise: scores in (3, 5] (65.4% of participants)

The vulnerability-fixing assignment asked students to improve the security of a web application by:
1. fixing vulnerabilities from a peer-generated report and
2. a predefined list based on OWASP Top 10 2021 and associated CWEs.

Students submitted both:
- the application changes, and
- a report describing each fixed vulnerability: its location, selected fix, and implemented security control.

What “usage patterns” meant in practice

The team coded each participant’s ChatGPT conversations using a semi-open coding procedure. They began with an initial set of codes, added new codes when fresh patterns emerged, and resolved disagreements via consensus between coders.

They then transformed codes into binary indicators P1–P14, meaning: for each pattern, a participant either showed that behavior or didn’t. This setup let them measure:
- how frequent each pattern was, and
- whether each pattern correlated with grades.

Finally, they also computed a diversity variable:
- n_patterns_used = the number of distinct usage patterns observed for a participant.

That diversity number is the key metric behind the main finding later on.

Usage strategies students used—and why expertise didn’t split the patterns

One of the more reassuring results for educators is that prior cybersecurity expertise didn’t dramatically change the overall way students used ChatGPT.

Most frequent ChatGPT strategies were similar across expertise groups

The dominant pattern in both expertise groups was asking ChatGPT to generate an implementation for a specific mitigation:
- 90% of medium-low expertise students
- 87.5% of high expertise students

Other common behaviors in both groups included:
- refining AI-generated code using feedback or errors,
- asking for explanations of security controls,
- debugging,
- optimizing or improving existing code.

So, both groups tended to use ChatGPT for direct task support—less “seeking general study help,” more “help me fix this security problem.”

Descriptive differences did show up, but they weren’t clean divides

Some patterns appeared more often among high-expertise students, for example:
- refining generated code,
- explaining security controls,
- exploring alternative solutions,
- checking whether their code was secure.

Meanwhile, medium-low expertise students more often:
- optimized existing code, and
- tried to work on the problem before leaning on ChatGPT.

But the paper cautions heavily here: the expertise groups were small and imbalanced (10 participants in medium-low, 16 in high). That means even a few students can swing percentages. The researchers interpret these differences as descriptive, not as proof of systematic behavior differences by expertise.

This is the comparison the policy debate often misses

Instead of “good students use AI differently,” the real picture seems closer to:
- students across expertise levels use ChatGPT in broadly similar ways
- but performance differences may come from how broadly they engage, not from whether they use particular tactics

That sets up the central analysis: performance isn’t just about “whether” students used ChatGPT patterns—it’s about how they used them overall.

Which individual ChatGPT patterns linked to grades (and why none were definitive)

It’s tempting to search for a single “best prompt style.” This study looked for that—but the statistical story is more modest.

Pattern-by-pattern comparisons: descriptive positives, no robust single winner

The researchers compared vulnerability-fixing performance between students who used each pattern vs. those who didn’t. They first looked at mean grade differences, and they also ran exploratory Wilcoxon rank-sum tests where appropriate.

Here are a few of the most notable descriptive differences (positive means higher average grades for students who used the pattern):

• Generate code based on mitigation name: +26.5 grade points
  (but this pattern was imbalanced: 23 used it, 3 didn’t)
• Refine AI-generated code with feedback and errors: +15.1
• Modify AI-generated code to fit my needs: +10.9
• Explore alternative solutions: +8.39
• Other smaller positives: debugging errors, optimizing existing code, attempting the problem before relying on ChatGPT

A few patterns were near zero or negative, such as:
- Copy and paste without reviewing carefully: −1.77
(observed for only 4 participants)
- Compare multiple versions of code: −1.65
- Explain a security control: −0.50

Why “none remained statistically significant” matters

Before multiple-comparison correction, one pattern showed statistical significance:
- generate code based on mitigation name: p = .016
After applying Benjamini–Hochberg correction for multiple comparisons, no single pattern remained statistically significant.

This doesn’t mean those patterns don’t matter—it means, in this exploratory study with small sample sizes and multiple tests, it’s hard to pin performance to any one behavior confidently.

So the researchers didn’t declare a “magic pattern.” Instead, they shifted attention to a broader measure: usage diversity.

The big result: more diverse ChatGPT use correlated with better secure fixing performance

If you want one takeaway that can shape how you teach secure engineering, it’s this: students who used a wider range of ChatGPT interaction patterns tended to perform better.

Usage diversity (n_patterns_used) showed a consistent positive association

The study operationalized diversity as the number of distinct coded usage patterns a student displayed.

In the one-sided Spearman correlation:
- ρ = .375
- p = .030

That’s a moderate positive association: higher diversity → higher grade, though it doesn’t explain everything (the scatterplot shows plenty of variability).

This relationship persisted even after controlling for prior cybersecurity expertise

They then fitted an exploratory linear regression model:
- Outcome: grade
- Predictor: n_patterns_used
- Control: cybersecurity expertise group (medium-low coded 0, high coded 1)

Key results:
- Diversity coefficient: b = 3.11, SE = 1.37, p = .033
- interpretation: each additional distinct ChatGPT usage pattern corresponded to an estimated +3.11 grade points, holding expertise constant
- Expertise group coefficient: b = 3.71, SE = 5.76, p = .526
- i.e., expertise alone wasn’t a clear predictor in this model
- Model fit: R² = .21
- Overall model test: F(2,23) = 3.06, p = .067 (not below .05)

So, while the model doesn’t “solve” prediction by itself, the diversity signal stays visible.

A practical interpretation: diversity looks like active problem-solving

What might “diverse usage” mean in student behavior? Likely a combination of:
- generating candidate fixes,
- asking for explanations,
- debugging errors,
- refining and adapting code,
- exploring alternative mitigations,
- and generally iterating instead of treating ChatGPT like an oracle.

This aligns with the paper’s argument: performance seems related less to any single mode of prompting, and more to whether students integrate ChatGPT into an active workflow.

Importantly, the study is observational, so the authors do not claim causality. Still, as a hypothesis, it’s strong and very actionable.

What educators can do today: designing secure engineering prompts for reflective use

Here’s where the research becomes useful beyond academic discussion. Since the result points toward diversity and task-oriented engagement, the natural next step is instructional design.

Stop rewarding “just the fix”; start rewarding evidence of evaluation

In security education, a fix that passes functional tests can be insecure. So students should be expected to verify security impact, not just produce working code.

The paper suggests guiding students to use ChatGPT for:
- targeted help with specific security controls,
- asking for explanations of proposed mitigations,
- checking whether generated code actually addresses the vulnerability,
- comparing alternative mitigations,
- and iterating with debugging/refinement.

In other words, ChatGPT should be a thinking partner, not a copy machine.

Build assessment criteria that reward reflective engagement

A practical approach is to update rubrics to reward:
- documentation of how suggestions were evaluated,
- adaptation and testing steps,
- reasoning about why the chosen fix mitigates the threat in the relevant context.

That sort of grading aligns with the study’s core finding: students who used a broader set of task-related patterns tended to get higher grades.

Compare “binary AI policy” vs “interaction-aware guidance”

This research implies a different posture than “allow vs ban.” Consider this contrast:

The paper’s discussion explicitly argues that simply permitting or prohibiting ChatGPT is less productive than teaching critical use, especially in security contexts.

Key Takeaways

• Students used broadly similar ChatGPT strategies regardless of whether they had medium-low or high prior cybersecurity expertise (e.g., asking for mitigation implementations was ~90% vs ~87.5%).
• No single individual ChatGPT usage pattern showed robust statistical significance after multiple-comparison correction, even though several patterns had descriptively higher grade differences.
• The strongest signal was ChatGPT usage diversity: the number of distinct interaction patterns (n_patterns_used) correlated positively with grades (ρ = .375, p = .030).
• This diversity–performance association remained after controlling for cybersecurity expertise in an exploratory regression (diversity: b = 3.11, p = .033; expertise group not significant).
• For education today: don’t just manage tool access—teach students to use LLMs as part of a reflective, multi-step secure fixing workflow, and adjust assessment to reward evaluation, adaptation, and validation rather than copy-pasted “fixes.”

If you want, I can also turn these findings into a concrete classroom checklist (what to prompt, what to log, what to grade) tailored to an OWASP/CWE-style vulnerability-fixing assignment.