AI Portrait Detectors Beat Young Adults—But Not Calibration

AI Portrait Detectors Beat Young Adults—But Not Calibration

AI-generated portraits are getting dangerously convincing—good enough to pass as profile pictures, ads, and even “news-like” visuals. So it’s no surprise that vision–language models (VLMs) are being pushed into the role of AI-content detectors. But new research from Kim et al. finds a twist: the newest VLMs have overtaken young adults at spotting AI-generated faces, yet they still struggle with something humans do well—calibration (basically, knowing when they’re right vs when they’re guessing).

In this study, the authors benchmarked 19 frontier VLMs on the same set of 198 face portraits used in an earlier human study of 1,667 adults. Those 198 images included real photos and identity-matched AI recreations from two generators: ChatGPT-4o and Imagen 3. The human baseline is already surprisingly strong—about 85% correct overall—but it drops steeply with age. Then the model story rapidly escalates: in June 2026, the best VLMs only matched humans in their 20s–30s. By July 2026, two new releases jumped past that ceiling.

Why This Matters: “Can It Detect AI?” Isn’t the Real Question Anymore

This research is significant right now because we’re entering a moment where “AI detection” is being productized—embedded in moderation tools, browser extensions, platforms, and reporting workflows. The headline users want is simple: Will it catch deepfakes? But the headline numbers can hide a nastier failure mode: a detector can be better than most people and still be bad for real deployments if it’s overconfident or has the wrong decision bias.

A concrete scenario you can apply today: imagine a platform that flags AI portraits for reduced reach or extra verification. If the detector is accurate but uncalibrated, it may confidently mislabel real people as AI (false alarms) or let AI slip through while still sounding certain. That’s not just an engineering annoyance—it’s a trust and fairness issue.

This study also builds on earlier work about human deepfake detection (where performance varies a lot by age and context) and on research showing models can rely on “shortcuts” or produce explanations that don’t actually reflect evidence. What’s new here is the insistence on a like-for-like benchmark: models are tested with the same kind of trials, the same stimulus pool, and the same scoring logic used for humans—so the “model vs human” comparison is actually meaningful.

What the Researchers Actually Measured: Accuracy, Bias, Sensitivity, and Calibration

To compare humans and machines fairly, the authors used a protocol designed to mirror the human experiment as closely as possible.

The exact face-portrait benchmark setup

Each VLM judged one image at a time as REAL or AI, and also returned:

• a confidence score from 0–100
• a one-sentence rationale

Models were prompted with a few-shot setup using four labeled practice images (2 real, 1 ChatGPT-4o, 1 Imagen 3), just like humans got during the practice block.

Humans and models were scored using balanced accuracy to match the session mix:

This is important because a model can look “good” by over-calling one class. Balanced accuracy punishes that.

The key comparison metrics beyond “% correct”

The authors didn’t stop at accuracy. They used a signal-detection-style view that separates two ideas:

• Sensitivity (called d′): how well the model separates real vs AI evidence
• Response bias (called c): whether the model leans toward answering AI or REAL

They then checked confidence calibration: do higher confidence scores correspond to being correct more often?

Here’s the core “calibration gap” finding in plain language: humans stay near-neutral decision bias across ages, but models can drift to extreme biases, and their confidence doesn’t reliably indicate correctness—especially in the earlier cohort.

The June-to-July Breakthrough: When the Ceiling Finally Broke

The study spans two waves of model releases, using the same benchmark protocol.

June 2026: Models match young adults, but don’t surpass them

In the June cohort (14 models), the top performers reached roughly the human range for people in their 20s–30s, but were not clearly better on average. The best single-pass balanced accuracies were around 87.1% (for gpt-5.5 and qwen3-vl-235b) and 86.7% (for claude-sonnet-4.6), which the paper reports as statistically indistinguishable from the human 20s–30s mean (~88.5% / 87.7% depending on the exact slice).

In other words: the “young adult ceiling” held in June.

July 2026: Two model releases jump above the ceiling

Then comes the key result: in the July cohort (5 models released about four weeks later), the ceiling breaks.

• gpt-5.6-sol: 92.8% single-pass balanced accuracy
  • and 92.1% average across five draws of few-shot examples
• claude-fable-5: 88.6% single-pass balanced accuracy
  • with perfect AI detection on what it saw (132/132 AI images)
  • but lower real-photo accuracy, leading to balanced accuracy of 91.9% on draw-averaged performance

The paper also reports sensitivity (d′) that exceeds young adults:

• d′ up to 3.4 for the best models
• young adults are around ≈2.4 (and older groups drop)

Model performance ranking isn’t “bigger model = better detector”

One of the more “don’t assume” results: model price and recency didn’t guarantee detection skill.

The paper notes cases like:

• claude-opus-4.8 scoring 64.0%, far below its sibling
• kimi-k3 (widely reported as strong on coding benchmarks) landing 12th of 19 at 80.9%
• the flagship not necessarily beating mid-tier versions

So this isn’t a general “capability translates across domains” story—it’s more specific than that.

Why Accuracy Alone Can Mislead: Instability and Example Sensitivity Still Exist

A major issue with leaderboards is that a single deterministic evaluation hides how fragile a system can be. The authors addressed that directly.

Few-shot examples still change decisions (about 1 in 4 images)

Even with temperature 0, models can be sensitive to the few-shot examples provided.

The authors reran the full benchmark five times per model, each time drawing a different set of labeled practice identities (still 2 real, 1 gpt-4o, 1 Imagen 3, excluding the drawn identities from the test set).

Across the 19-model fleet, the flip rate—the fraction of images where a model changed its answer at least once—was:

• 25.7% overall
• 24.7% for strict REAL↔AI reversals
• 23.3% strict reversals due to variation in the chosen few-shot draws alone

That’s basically: roughly one image out of four can flip.

Instability doesn’t vanish in July’s winners

The July leaders stayed high—but not perfectly stable.

• gpt-5.6-sol ranged down to a level near June performance on one draw (~84.5% on the worst draw reported)
• claude-fable-5 also shows draw variance, and interestingly the baseline draw was reported as its worst pass

So yes, the detectors improved—and dramatically—but single-pass results can overstate reliability.

The Calibration Problem: Better Seeing, Worse Deciding (and Sometimes Overconfident)

This is the part that really matters if you plan to deploy detectors in the real world.

Humans: near-neutral bias, and confidence tracks accuracy

Humans are described as nearly unbiased in their decision criterion (c ≈ 0) across all ages. Sensitivity (d′) declines with age—but their decision threshold doesn’t drift into extreme “always AI” or “always real” territory.

Also, human confidence tracks accuracy with a reported between-participant Spearman correlation of ρ = 0.379 (n = 1,667).

Models: sensitivity improves, but bias gets extreme

Models varied wildly in response bias (c), spanning:

• from c = −1.10 (very AI-leaning)
• to c = +1.45 (very REAL-leaning)

The new leaders landed near opposite biases:

• claude-fable-5: c = −0.97 (strongly AI-leaning)
• gpt-5.6-sol: c = +0.44 (leans toward REAL compared to humans)

And here’s the crucial “not overtaken” insight from the paper:
humans sit near zero bias at every age, while only a few models happen to approach human balance.

Confidence calibration improved in July—but overconfidence still happens

In June, most models were overconfident:

• mean stated confidence could exceed balanced accuracy by up to +29 pp

In July, the two best leaders improved calibration substantially:

• gpt-5.6-sol: gap about +0.1 pp
• claude-fable-5: about −5.7 pp (slightly underconfident)

They also improved “resolution,” meaning confidence better separated correct from wrong answers.

But calibration isn’t the whole story: even a top model could be confidently wrong. The paper includes an example where a June top-3 model accepts an AI portrait as natural at confidence 95—showing how persuasive confidence can still be when the decision criterion is off.

Why balanced accuracy can still be “human-like” despite super discrimination

This is the subtle logic at the heart of the paper:

• claude-fable-5 discriminates strongly (d′ is very high)
• but its bias causes it to call many real photos as AI
• balanced accuracy then lands near young-adult levels rather than “destroying” the humans

So you can think of it like this:
machines got better at smelling the difference, but they chose a different “when to accuse” line than humans did.

Do Models Use Real Evidence? Mostly They Echo Verdicts

The authors also analyzed the 3,738 model rationales (one sentence per prediction) using a cue taxonomy (15 categories). They found vocabulary overlap with human reports—models often mention things like:

• over-smoothness / texture
• skin texture
• lighting/shadows
• uncanny or “too-perfect” appearance
• natural/realistic impression

But the punchline is that the cues often function as verdict markers, not independent evidence.

“Over-smooth” can mean AI—or can mean you got it wrong

The cue validity flips depending on the true class.

For example, “over-smoothness” is associated with:

• +24.4 pp accuracy on AI images when invoked
• but −77.6 pp on real photographs when invoked

Similarly, “natural/realistic” shows a mirror pattern.

That means the model’s explanation is often:
“I answered AI, so I’m describing it in an AI-ish way.”

Only hair-related cues showed weak positive value in both classes.

Humans admit uncertainty; models basically don’t

In human checklists from the earlier study, about 49.1% endorsed “feeling/intuition”—an explicit uncertainty marker. In model rationales, expressions of uncertainty were essentially absent (0%).

So model explanations look confident even when confidence/calibration isn’t guaranteed.

This also aligns with broader findings (the paper cites work suggesting model explanations can be unfaithful to the true basis of decisions). The comparison here just makes it concrete: explanations track the verdict rather than reliably pointing to the evidence that drove it.

Key Takeaways

• Yes, frontier VLMs have overtaken young adults on detecting AI-generated portraits (on this benchmark).
  • The July models gpt-5.6-sol and claude-fable-5 exceed young-adult means in balanced accuracy and sensitivity (d′ up to ~3.4).
• No, calibration hasn’t been overtaken.
  • Humans keep decision bias near c ≈ 0 across ages.
  • Models spread widely (c from −1.10 to +1.45), and the leaders still have non-human biases.
• Reliability is still fragile.
  • Changing the labeled few-shot examples flips about 25.7% of answers across the model fleet (roughly one in four images).
• Confidence helps, but it’s not a safety guarantee.
  • July leaders show improved calibration (e.g., gpt-5.6-sol gap ~+0.1 pp), but confident mistakes still occur.
• Rationales often function as verdict narration, not evidence.
  • Cues like “over-smooth” and “natural/realistic” can flip sign depending on whether the image is actually AI or real—suggesting the explanation is often aligned with the model’s answer rather than exposing trustworthy forensic signals.
• Practical implication today: if you’re deploying detection, don’t treat “model confidence” as an automatic risk score. You likely need calibrated thresholds, uncertainty handling, and monitoring for bias-driven false alarms.

If you want, I can also turn this into a short “deployment checklist” (what to log, how to set thresholds, and how to stress-test calibration) based directly on the study’s failure modes.