AI scripts from any model share the same security holes

AI scripts from any model share the same security holes

LLM coding tools are getting scary-good at turning a plain-English request into working Python. But new research from Kahn & Hastings (arXiv:2607.20713) digs into a question that matters way more than “does it run?”: do these AI-generated automation scripts also contain exploitable security vulnerabilities when non-experts deploy them inside real enterprise workflows?

The study compares three popular coding assistants—ChatGPT, Microsoft Copilot, and Google Gemini—by using identical prompts across three automation task domains. Then, rather than just checking if the scripts functioned, the researchers had Claude Code perform a standardized vulnerability review. Vulnerabilities were classified and scored using CVSS v3.1, and mapped to OWASP Top 10:2021 and MITRE ATT&CK so the results connect to real-world risk and attacker behavior.

The headline result is blunt: every single script contained exploitable vulnerabilities, and the most important pattern wasn’t “which model is worst.” It was that the same vulnerability types keep showing up depending on the task category, not the tool.

Why This Matters: LLM Security Risk Is Becoming an “Automation Tax”

This research is significant right now because organizations are past the “pilot stage” of LLMs in many teams. People aren’t only using AI to write app features—they’re using it to generate small automation scripts: web scrapers, email senders, file sorters, workflow helpers. Those scripts often bypass the careful security review you’d expect for production code. The paper frames the real user as an office worker, not a developer, and that mismatch is exactly where risk multiplies.

Here’s a concrete scenario you can apply today. Imagine a team member uses an LLM to build an automation that:
- scrapes product pages,
- pulls data into a report,
- emails templated updates,
- and moves files into shared folders.

Those scripts touch the exact places attackers love: external URLs, internal network resources, email infrastructure, and shared drives. Even if each script looks “harmless,” the research shows predictable vulnerability patterns—like SSRF (server-side request forgery) in scrapers and path traversal in file workflow automation—that can turn a convenience tool into a foothold.

Compared to earlier AI code research, this study adds a more operational angle. Previous work has shown that LLM-generated code can be vulnerable at scale, but Kahn & Hastings focus on a setting that matches how the tools are actually used: non-experts run the first working output. And they do a cross-model comparison under controlled conditions (same prompts, same tasks), which makes the findings more actionable: the risk is structural.

That’s the pivot organizations need: don’t just decide “which AI tool to trust.” Decide whether LLM-generated automation code should run without review at all.

What the Researchers Actually Measured: Nine Scripts, Three Models, One Security Lens

This study collected code from three LLMs using the same prompts, for three realistic automation tasks, and then applied a standardized vulnerability assessment.

The task domains that mirror real work

The researchers selected these three task categories after initially generating candidate tasks from the models and narrowing down options:
1. Web scraping & data extraction
2. Email & document automation
3. File & workflow automation

These are common automation requests because they’re useful and easy to describe in everyday language—even for people who don’t think in terms of injection flaws, validation gaps, or SSRF.

The exact comparison design (the key experimental control)

The core idea: keep everything constant except the model.

The researchers generated 9 Python scripts total (3 models × 3 task domains). No manual edits were made before review—important, because the paper targets the real-world case where users deploy the first functional code they get.

How vulnerabilities were reviewed and scored

• Vulnerability review tool: Claude Code
• Output handling: results were deduplicated into a consolidated list
• Initial discoveries: 45 vulnerabilities reported, reduced to 17 unique vulnerability classes after deduplication
• Scoring method: CVSS v3.1 (to compare severity/exploitability)
• Framework mapping:
  • OWASP Top 10:2021 (to connect to common web/enterprise risk categories)
  • MITRE ATT&CK (to connect findings to attacker tactics and techniques)

One important nuance: the scoring was treated as a reasonable worst-case assumption and does not include environment-specific mitigations. In other words, CVSS here is about “if this is reachable, how bad could it be,” not “how safe is it in your particular deployment.”

The Biggest Pattern: Every Script Had Vulnerabilities—And Models Didn’t Matter Much

Let’s get to the results that drive the paper’s main conclusion: vulnerabilities aren’t a “bad-model” problem.

Cross-model overlap shows convergence on the same weak patterns

The study found substantial overlap in vulnerability classes across models. Specifically:
- 9 of 17 vulnerability classes showed up in code generated by all three models.
- 14 of 17 vulnerability classes appeared in code from at least two models.
- The gap in total weighted CVSS scores across platforms differed by less than 10%.

That matters because it undercuts a common instinct: “We’ll just switch to the safest model.” The researchers basically show that these tools converge toward similar insecure implementation patterns once asked to build the same kind of automation.

Where the vulnerabilities cluster by task category

The paper highlights repeated patterns that appear across multiple domains and models. Examples include:
- Path traversal appearing in scraper, email, and file watcher code
- Overly broad exception handlers appearing across email senders
- Symlink vulnerabilities appearing in all watcher scripts
- Template injection appearing in each email sender
- SSRF appearing in each scraper script

If you’ve ever seen “copy-paste bugs” spread through codebases, this is similar—but at the level of security design defaults. Different models, same prompt, same task—same kinds of mistakes.

Comparison of total risk between models (high-level)

The paper reports that weighted CVSS totals were tightly clustered, with less than 10% difference between models. That supports the key conclusion: the risk is not tied to any particular model, but to the task category and how these scripts are typically written.

How These Vulnerabilities Turn into Real Attacks: SSRF, Traversal, and Injection as Entry Points

One of the most useful parts of this research is how it ties code weaknesses to attacker workflows. They didn’t just say “there are vulnerabilities.” They mapped them to OWASP and MITRE ATT&CK so the findings connect to known exploitation paths.

Vulnerabilities found are “low-effort, high-impact”

The study argues the vulnerabilities aren’t hard to exploit under typical enterprise assumptions. Their assumptions include:
- attacker has user-level access (not admin),
- scripts are deployed in a typical enterprise-ish environment (not a hardened sandbox),
- scripts are trusted and already deployed,
- targets include shared drives and email-related systems.

Under those conditions, vulnerabilities like the following become practical:
- SSRF: a malicious URL passed to a scraper triggers internal requests
- Path traversal: config-controlled paths allow reading/writing outside intended directories
- Email header injection: attacker-controlled fields can alter outbound headers and redirect content/delivery
- Template injection: unsafe templating can let injected content impact generated email/document outputs

“It’s a chain” isn’t a cliché here—it’s a consequence of how scripts work

The paper emphasizes that attackers don’t stop at the first weakness. For example:
- SSRF can be used to probe internal services and help with credential discovery.
- Path traversal doesn’t just expose a file—if write access is possible, it enables persistence.
- A script that both logs sensitive data and moves files based on unvalidated input can create compound damage rather than a single isolated issue.

The risk isn’t just one vulnerability class—it spans multiple kill-chain stages

They map findings through the Lockheed Martin Cyber Kill Chain (and also provide five primary attack paths later in the paper). The take-home is that these automation scripts can cover multiple attacker phases:
- initial access (e.g., SSRF, traversal),
- execution (e.g., template injection, header manipulation),
- credential access (e.g., reading/logging secrets from config),
- lateral movement (once file system access exists),
- exfiltration/data access.

Crucially, the paper argues that many of these paths do not require privilege escalation or sophisticated techniques. The scripts inherit the permissions of the user or host they run on, which is often “enough” inside real organizations.

Prioritize Like a Security Team: Pareto Risk Shows Where to Start Fixing

If you’re responsible for security review or policy, the paper gives you something practical: not all 17 vulnerability classes matter equally.

Weighted severity reveals a “top tier”

They performed a Pareto-style analysis using:
- CVSS base score × number of models affected

Result: 11 of the 17 vulnerability classes account for ~80% of the total weighted risk.

The paper explicitly calls out that focusing remediation first on the “top tier” would reduce most exposure. It mentions high-impact categories including:
- Template injection
- SSRF
- Email header injection
- Path traversal

CVSS is useful—but it’s not the whole truth

The paper also warns against a simplistic read of CVSS:
- CVSS assumes worst-case exploitation conditions.
- Real risk depends on deployment details: network reachability, identity controls, and defense-in-depth.

So CVSS should be treated as a starting point for prioritization, not a final verdict. But the Pareto result still helps you decide what to do first when resources are limited.

What Organizations Should Do Next: Treat LLM Automation Code Like Untrusted Input

This is where the paper’s conclusion becomes a policy message.

The key recommendation: don’t ask “which tool,” ask “should this run”

The authors are clear: the security risk is systemic, not a model-brand issue. Because vulnerability overlap is high and weighted differences are small (under 10%), the “tool switching” strategy doesn’t solve the problem.

Instead, organizations should ask:
- Should LLM-generated automation code be deployed without review?
- Which categories of scripts are allowed to run, and under what restrictions?
- Do we have guardrails for SSRF-like behavior, path handling, template rendering, and email header building?

How this connects to OWASP and MITRE—so you can act, not just worry

Mapping to OWASP Top 10:2021 makes the findings usable for teams that already have defensive playbooks. The paper notes mapping into categories like:
- Injection,
- Broken Access Control,
- Server-Side Request Forgery,
- Security Misconfiguration,
among others.

Then MITRE ATT&CK mapping gives security teams a way to tie findings back to adversary tactics and techniques—useful for threat modeling, detection engineering, and prioritizing mitigations that actually disrupt attacker paths.

What to implement in practice (the “review gate” mindset)

Based on the study’s findings, the most realistic operational move is to require a security gate for LLM-generated automation code, especially for:
- scrapers accessing external URLs,
- email automation that builds headers/templates,
- file/workflow automation that uses config-controlled paths.

That gate doesn’t necessarily mean “hire a security engineer for every script.” But it does mean:
- enforce input validation and output encoding requirements,
- restrict network destinations and internal reachability,
- lock down filesystem operations to safe directories,
- treat templates and email fields as untrusted data that must be handled defensively.

Key Takeaways

• Every one of the 9 evaluated scripts contained exploitable security vulnerabilities. There was no “safe output” case in this experiment.
• The same vulnerability types kept recurring across models.
  • 9/17 vulnerability classes appeared in all three models
  • 14/17 appeared in at least two
• Risk was not tied to a specific model. Weighted CVSS totals differed by less than 10% across platforms, supporting the idea that the weakness is tied to task categories and typical automation patterns, not the brand of LLM.
• High-impact vulnerabilities dominated exposure. Using weighted Pareto analysis, 11/17 vulnerability classes account for ~80% of total weighted risk, with top priorities including template injection, SSRF, email header injection, and path traversal.
• Functional code isn’t security code. The scripts met their tasks (scrape, email, sort), but still introduced attacker-friendly entry points that map to OWASP Top 10 and MITRE ATT&CK.
• Practical next step: stop asking “which LLM is safest” and start asking “should LLM-generated automation code run without review,” plus add category-specific guardrails for scraping, emailing, and filesystem workflow scripts.

If you want, tell me what kind of automations your team runs (scraping, email, file movement, report generation, API calls, etc.), and I’ll translate these findings into a simple policy checklist and review questions you can use with non-expert users.