API scores don’t predict chatbot behavior—careful!

API scores don’t predict chatbot behavior—careful!

Introduction

If you’ve ever trusted an LLM “benchmark score” to predict what you’ll experience in a real chatbot, this new research is going to make you pause. The core idea in AI evaluation is simple: run a model on a set of tests, publish the results, and let those numbers guide decisions. But what if the numbers don’t actually travel well from the place they’re measured (often an API) to the place people use them (a web chat interface)?

That’s exactly what Jennifer Wang, Joachim Baumann, Daniel E. Ho, and Sanmi Koyejo investigate in their new paper, API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces. Their focus is “context validity” — whether benchmark outcomes measured under one evaluation setup generalize to another. In this case, the two contexts are: API calls vs deployed chatbot interfaces (the kind you interact with as a user).

They audit multiple frontier models across ChatGPT, Claude, and Gemini, using the same prompts and applying the same scoring pipeline—yet they still find meaningful divergences in accuracy, consistency, and even rankings. In other words: benchmark results may be real, but they’re not always the same “real” when you move across access surfaces.

Why This Matters

This matters right now because benchmark numbers have effectively become a coordination system for the whole ecosystem. Researchers, product teams, journalists, and procurement folks use them as a shorthand for capability and risk. If those numbers drift depending on whether you test through an API or a chatbot UI, then the shorthand is sometimes misleading—and the mismatch can be big enough to affect decisions.

Here’s a scenario where this bites today: suppose a company is screening models for customer support automation. A vendor might cite API benchmark performance and conclude “this model is strong on factuality and reasoning.” But when the same model is embedded into the company’s actual workflow through a chat interface (with whatever system prompt, routing, tool policy, or UI-time processing the vendor uses), the model’s behavior can change—sometimes in accuracy, sometimes in reliability, and sometimes in the ordering between “which model is best.”

This research builds on a long line of work arguing that benchmarks are sensitive to evaluation details (prompt formats, scoring rules, and so on). The newer twist is that even if you keep the benchmark items and scoring procedure constant, the access surface itself (API vs interface) can reshape outcomes. That’s a structural problem, not just a tweakable one.

What the Researchers Actually Measured: API vs Interface Across Multiple Benchmarks

The study asks a clean question: if you send identical prompts, will the benchmark results match when the prompt goes through the API versus when it goes through the user-facing chatbot interface?

To do this at scale, the authors run controlled “black box” evaluations across seven systems spread across three providers: ChatGPT, Claude, and Gemini. For each interface system, they try to match it to the closest available documented API model identifier. (Important limitation: providers don’t always reveal the exact checkpoint behind the interface, so “matching” is necessarily approximate.)

They evaluate nine benchmarks grouped into two categories:

General capabilities (leaderboard-style tasks)

They start with six common leaderboards used in model release notes—things like math, reading comprehension, and multiple-choice reasoning:

• ARC (ARC Challenge)
• GSM8K
• HellaSwag
• MMLU
• TruthfulQA (TQA)
• WinoGrande (WG)

Running the full suite through chatbot interfaces is too expensive and slow, so they use Metabench, a distilled sparse set. Instead of 28,659 items, they evaluate 858 highly informative items for this general-capability group.

User-facing risks (things that matter in real deployments)

They also audit behaviors tied to user-visible harms:

• Social bias: BBQ (Bias Benchmark for Question Answering)
• Sycophancy: AITA-NTA (AITA) using the ELEPHANT setup
• Cross-domain reliability: AA-Omniscience (AA-Omni.)

Sampling details matter because they affect reliability:
- For BBQ and AA-Omniscience, they sample 200 randomly chosen items.
- For AITA, they sample 100 original–flipped pairs and score both sides in the same way.

Their evaluation method: same prompts, controlled scoring

For each benchmark item, they issue the same prompt to both:
1. the model API
2. the model interface

Then they normalize the output:
- Interface output is stripped to plain text so UI formatting doesn’t affect scoring.
- They use a single benchmark-specific extraction and grading pipeline for both conditions.

They even run five independent trials, rotating across accounts and using fresh browser state each time (no chat history, no persistent cookies). Data collection spans March 6 to May 24, 2026, and they cap throughput at 150 queries per three-hour window with a two-hour cooldown after rate limiting to avoid triggering protective automation defenses.

How they measure “reliability” beyond just average accuracy

They don’t only report accuracy. They also look at:
- Test–retest agreement: if you run the same item multiple times, do you keep getting the same answer? This is crucial because chat interfaces can be more variable.
- Rank stability: do model rankings stay consistent across access surfaces?

Results That Should Change How You Interpret Benchmarks

The headline finding: API evaluations score higher and are more consistent than interface evaluations on average, but the story isn’t “API always wins.” The direction and size of the gap varies by model and benchmark.

Average accuracy gap: API higher by 3.4 percentage points

Across the audit, API accuracy is consistently higher than interface accuracy.

• Average gap: 3.4% (SE = 1.11)
• This holds statistically even after controlling for benchmark, system, and item clustering (p = 0.002)

At the system level, all seven systems show significantly higher accuracy through the API (all p < 0.001).

When the gap is small vs when it’s huge

The magnitude varies a lot. Some pairs show modest shifts; others show large differences.

Here’s the pattern the paper highlights: the largest differences appear in certain user-facing or more sensitive benchmarks.

So even though the average gap is 3.4%, there are cases where the interface difference is large enough to look like a different capability altogether.

API gaps can rival differences between API model versions

A really telling comparison: the authors measure how much performance changes between two API-only model versions—and then compare that to the API-to-interface gap for the same model generation.

Across the same benchmarks via the API:
- Mean absolute difference between GPT 5.3 Instant and GPT 5.4 Instant: 4.5% (SE = 1.57, p = 0.021)

But for some interface cases, the interface gap for GPT 5.4 Thinking is larger than that 4.5% version gap. For example, on WinoGrande, GPT 5.4 Thinking shows:
- API–interface gap: +19.0%
- API-only model-version difference: about 9%

That means the access-surface effect can be bigger than the difference between two generations—which is a pretty wild implication for how people casually read leaderboard charts.

Rankings don’t transfer reliably

If average accuracy differences already complicate things, ranking changes make it even harder to use API benchmarks as a proxy for deployed systems.

Across the nine benchmarks:
- Spearman correlation of rankings: ρ = 0.52
- This is “partially carries over,” not “basically the same.”

The paper also shows a crucial detail: rank correspondence is weakest when systems are tightly clustered—exactly where fine leaderboard distinctions matter.

For example:
- On MMLU, score range is about 92–96% and rank correlation drops to ρ = 0.11
- On AA-Omniscience, score range is wider (12–63%) and rank correlation is ρ = 0.96

They even observe rank inversions within the same provider:
- On BBQ, GPT 5.4 Thinking outranks GPT 5.3 Instant via the API (92.8% vs 91.5%), but flips via the interface (90.8% vs 93.9%).
- On WinoGrande, Claude Sonnet 4.6 is ahead via API (92.4% vs 87.3%) but Haiku 4.5 edges ahead via interface (89.0% vs 88.3%).

Consistency is worse in interfaces

This isn’t just about “who scores higher.” The interface behavior also changes more from run to run.

Average test–retest agreement:
- API: 96.9%
- Interface: 94.8%
- Gap: 2.1 percentage points (SE(Δ) = 0.17, p < 0.001, 95% CI [1.8, 2.4])

The gap appears in all seven systems. Three models reach p < 0.01 for this reliability difference (including ChatGPT Instant, ChatGPT Thinking, and Gemini Thinking). The smallest reliability gap is for Claude Opus 4.6 (about +0.7%, q near significance).

Why the Gap Exists: “API Controls” Can’t Fully Recreate Interface Behavior

After seeing the divergences, the authors test a natural hypothesis:

Maybe interfaces are just using hidden “extra settings”—like a system prompt, different decoding parameters, or reasoning constraints. If that’s true, then the API benchmark should match once you reproduce those interface settings.

They try two kinds of controls.

Control 1: Add approximated system prompts to API calls

Interfaces often run with hidden system prompts. The authors approximate these using publicly circulated prompt templates for each provider, then prepend them to API requests.

Result:
- Test–retest agreement actually drops slightly relative to baseline API:
- Prompted API: 95.4% vs baseline API: 97.2%
- It moves closer to the interface, but doesn’t fully match:
- Interface test–retest: 94.8%
- The remaining agreement difference becomes statistically insignificant (Δ = +0.5 pp, p = 0.413)

That sounds promising for consistency—but it fails for accuracy.

Accuracy gap does not disappear:
- Mean absolute interface–API gap decreases only from 6.5 pp to 6.4 pp
- That reduction is not significant (p = 0.94) across 45 system–benchmark pairs where prompts are available (the paper notes prompt availability only for a subset of models)

So adding prompts can align some aspects of behavior (especially variability), but it doesn’t close the main accuracy gap.

Control 2: Sweep sampling and reasoning parameters in the API

Next, they vary exposed API parameters and check whether they can replicate interface behavior.

They run targeted parameter sweeps on two benchmarks that show significant interface–API gaps:
- BBQ
- HellaSwag

Sampling controls:
- temperature in {0.0, 0.5, 0.7}
- top-p in {0.9, 0.95}

Reasoning controls (model-specific):
- For GPT-5.4: reasoning_effort in {low, medium, high}
- For Gemini 3 Flash: thinking_level in {low, medium, high}
- For Claude Sonnet 4.6 and Haiku 4.5: budget_tokens in {1024, 4096, 16384}

What happens?
- Accuracy is remarkably stable across these sampling sweeps (all sweep ANOVA tests have p > 0.30)
- Reasoning budget shows a modest effect in one sweep:
- GPT 5.4 on HellaSwag shifts by about 2.2 pp
- But that’s still not enough to explain the broader interface gap (and it doesn’t affect reliability in a way that closes the mismatch)

Bottom line: even when researchers can manipulate “some” knobs, the interface’s behavior doesn’t reliably emerge from the API configuration space that external users can access.

What This Means for Researchers, Benchmark Authors, and Product Teams

The practical implication is uncomfortable but clarifying:

API benchmark scores are evidence about the API configuration—not necessarily about the deployed chatbot.

The paper explicitly argues that benchmark reports should document the access path and deployment context. If you’re a researcher running evaluations to support a claim like “Model X is more truthful,” you may be making a claim that doesn’t survive contact with how users actually access the system.

If you’re building evals for a real product

A product team can’t just take a leaderboard score at face value. If you care about user-facing behavior, you need either:
- interface-based evaluations, or
- API evaluations validated against interface behavior for your specific provider/model/version

Otherwise you risk shipping a system that performs differently once it passes through the hidden “middle layers.”

Why this is partly about “the middle layers”

The authors point out that there’s an inaccessible stack between a model endpoint and a UI response. Even if you match prompts and sampling, the interface may still include:
- routing logic (which variant gets served)
- retrieval and conversation context
- tool policies and safety post-processing
- other transformations that are not exposed in the API contract

This is the core ecological validity problem: you can test the model, but the user experiences the entire product pipeline.

A note on limitations (so you don’t over-interpret)

The paper’s own limitations are worth respecting:
- They can’t verify that the matched API model and interface model are the exact same checkpoint.
- They test only one subscription tier per platform.
- They use benchmark items rather than fully natural chat interactions; longer conversation dynamics might interact differently with memory and context.
- The audit covers three providers (and seven systems), so results may not generalize perfectly to all platforms or future releases.

Still, the consistency of the finding—divergence, variable magnitude, and stability/reliability differences—makes it hard to dismiss.

And the authors also release an open-source tool for interface auditing: interface-audit at their GitHub repo. That’s important because it lowers the barrier to doing this kind of measurement continuously, not just once as a research project.

Key Takeaways

• Benchmark scores don’t reliably transfer from API evaluations to deployed chatbot interfaces, even with identical prompts and shared scoring pipelines.
• On average, API accuracy is higher than interface accuracy by 3.4% (SE = 1.11), and this holds across all seven systems tested.
• The gap is not uniform: some benchmark–model pairs show enormous differences (e.g., AITA with +28.4% for Claude Haiku 4.5).
• Rankings can change across access surfaces (mean Spearman rank correlation ρ = 0.52 across nine benchmarks), including provider-level rank inversions.
• Interfaces are also less consistent: test–retest agreement is 96.9% (API) vs 94.8% (interface), a 2.1 pp difference.
• Trying to reproduce interface behavior using approximated system prompts and API-exposed sampling/reasoning controls does not reliably eliminate the accuracy gap.
• Practical rule of thumb: if your decisions depend on user-facing behavior, don’t treat API benchmark numbers as a perfect proxy unless you’ve validated the bridge for your specific provider/model/version.

If you want, tell me what you’re using benchmarks for (research claim, internal vendor comparison, product selection, safety evaluation, etc.), and I can suggest a practical evaluation plan that reduces this API-vs-interface mismatch risk.