Agentic Web Search: How Chatbots Decide, Query, and Attribute Facts

Agentic Web Search: How Chatbots Decide, Query, and Attribute Facts

Conversational AI isn’t just “answering questions” anymore—it’s increasingly acting like a Web search agent, deciding when it needs the internet, what to ask, and which sources to rely on. That might sound straightforward, but new research from Amani et al. (arXiv:2609.19244) shows the full story is messy—and importantly, different across platforms.

The authors run a rare kind of end-to-end study. They look at real user interactions (an “invivo” setup) with four major chat platforms—ChatGPT, Claude, Grok, and DeepSeek—and pair that with controlled experiments (an “invitro” setup) where the same kinds of prompts are sent through the platforms’ APIs. That combination lets them examine the entire lifecycle of agentic search: search decisions, query strategies, and response grounding/citations.

What they find will make you rethink “the bot looked it up, so it must be right.” Web-search behavior varies a lot by model; more searching doesn’t always mean better answers; search engines return biased and domain-preferred results; and—perhaps most concerning—some claims in final responses are supported by uncited search results (and some claims are still ungrounded).

Why This Matters: Agentic Search Is Everywhere, but Reliability Isn’t Plug-and-Play

This research is significant right now because “tool use” is becoming a default feature. If your assistant can search the Web, it can also answer time-sensitive questions, cite sources, and look more authoritative—at least on the surface. But the paper makes the key point: agentic search is a chain of decisions, and each link can fail or skew outcomes.

A concrete scenario where this shows up today: imagine using a chatbot at work to draft a policy brief or health-related summary. If the assistant decides it needs Web search, it might pull information from a small set of domains that its platform’s search system prefers. Even if the response contains citations, those citations may not reflect a comprehensive view of the Web, and some content used to produce the answer may come from sources that never get cited. That’s not just a transparency issue—it affects how you verify, audit, and trust what the assistant produced.

This work builds on previous AI research that often studied retrieval in isolation (or focused mainly on whether retrieval happens at all). Instead, it treats the assistant like a full search lifecycle system—when it searches, how it searches, and what it does with the retrieved evidence. That’s a big step forward, and it directly supports safer design: evaluation can’t stop at “did it retrieve?” It must also ask “did it attribute correctly, and did the chosen evidence actually support the claims?”

What the Researchers Actually Measured: The End-to-End Lifecycle of Web Search Agents

To study agentic Web search properly, the authors break the lifecycle into three stages:

1. Web-search decision: does the agent decide it needs external knowledge beyond its internal (parametric) knowledge?
2. Query formulation: what Web queries does it generate, and does it refine them across iterations?
3. Response generation and grounding: how does it synthesize results into claims, and how well are those claims supported by cited or retrieved evidence?

They then use two complementary dataset strategies:

• Invivo data (real user traces): 171,264 conversations from 613 users across ChatGPT, Claude, Grok, and DeepSeek. Each conversation turn includes user prompts, Web query traces, search results, cited URLs, and generated responses. (The donated data itself isn’t publicly released for privacy reasons, but the authors release mock data formats and analysis code.)
• Invitro experiments (API-controlled replay): they run the same style of prompts through the platforms’ APIs to compare Web-search behavior under identical inputs. They use 1,000 user prompts (only the first user message), filtering out personal information via platform API controls.

The key methodological win: the invivo setting tells you what agents do in the wild, while invitro isolates how behavior changes when the same prompt is used across model families. And importantly for this paper: they can study model-family differences, not just platform differences.

If you want the core artifact that ties the paper together, the authors link to their code repository for the analysis pipeline: GitHub: AgenticSearchLens.

How Agents Decide to Search (and Why “More Searching” Isn’t Automatically Better)

Let’s start with the first stage: the moment a chatbot decides whether to call Web search at all. Humans can recognize when they’re unsure; the model has to do something similar internally—judge whether its built-in knowledge is enough.

The paper observes that Web-search invocation increases over time across platforms (shown for ChatGPT, with trends described more broadly). But the frequency also depends on topic: for example, travel prompts trigger Web search more often than health prompts.

The most striking part is how much the models disagree with each other in a controlled setting. In invitro (same 1,000 prompts), Web-search calls vary dramatically:

So yes—some agents behave like cautious researchers, while others behave like confident generalists. That alone matters for user experience (speed, cost, perceived thoroughness), but the paper goes further: it tests whether search invocation actually helps.

They evaluate response quality using three metrics (each on a 1–5 scale):
- factuality (correctness)
- completeness (coverage)
- relevance (does it address what the user asked?)

And they compare three groups:
1. prompts where the agent did call Web search
2. the same prompts with Web search disabled
3. prompts where the agent did not call Web search

Three big observations stand out:

• Web search often improves quality when the agent decides to use it, but the size and even the direction of improvement vary by model. The authors report clear benefits for GPT-5.3-chat, Grok-4.3, and DeepSeek-v4-flash, while Claude Sonnet 4.6 shows at best mixed effects despite searching a lot.
• When agents choose not to search, answers can score higher across all quality measures. That’s a sign that LLMs aren’t randomly deciding—they can be pretty good at recognizing when parametric knowledge is sufficient.
• Even when prompts seem to require Web search, quality is still lower than for prompts that don’t require Web search, suggesting Web retrieval doesn’t fully “patch” internal knowledge gaps.

The takeaway: search is not a universal upgrade. A platform can “search more,” yet still not systematically produce better answers.

How Agents Build Search Queries—and What That Reveals About Strategy

Once an agent decides to search, it has to turn a verbose user prompt into something a search engine can handle. The paper shows this is where agents become more “operations-focused” than you might expect.

Multiple queries, fan-out vs deep iteration

Unlike typical human search (often a single query with manual tweaks), conversational agents can orchestrate multiple queries. The authors analyze both:
- how many queries happen per prompt
- whether they happen in parallel (fan-out) or across sequential iterations

In the invivo data, the median number of Web queries per prompt ranges from 2 (Claude) to 4 (Grok), with maxima sometimes exceeding 10.

They also observe a behavioral difference:
- ChatGPT, Grok, and DeepSeek tend to use fan-out (parallel queries within iterations),
- while Claude tends to generate one query per iteration, leading to deeper multi-iteration search.

In invitro experiments with the API models, the pattern changes somewhat:
- Only GPT-5.3 issues fan-out queries.
- Grok, DeepSeek, and Claude issue only one query per iteration.

This suggests harness instructions and model-level tool policies matter a lot—but the overall lesson is consistent: there’s no single best query strategy. Different systems explore the search space differently.

Query rewriting: compressed, context-driven, and increasingly specific

The paper also measures query length and specificity. Agents don’t usually copy the full user prompt into a query. Instead, they compress it:
- nearly 80% of user prompts contain more than 20 terms
- but almost all generated Web queries contain fewer than 10–15 terms

Even more interesting: as iterations progress, new query terms increasingly come from previous search results rather than only from the latest prompt. The authors track term provenance and find:
- first-iteration query terms come primarily from the latest prompt (about 30%) plus conversation history (about 80%),
- later iterations introduce more terms tied to what the agent saw in retrieved sources.

On top of that, agents steadily make queries more specific across iterations—adding constraints like:
- exact dates
- geographic signals
- named entities

The authors quantify specificity across time/geography/entity dimensions (using a Likert-style 1–5 scoring scheme) and find it increases consistently.

Analogy-wise: think of it like narrowing down a messy library search. Early on you use broad terms (“best laptops for students”). Over iterations you stop asking “best laptops” and start asking “best 14-inch laptops for computer science students 2024 with 16GB RAM,” based on what you found in earlier passes.

Platform search engines introduce domain preferences

Here’s a subtle but huge point: the “Web search” tool isn’t a neutral Google-like interface. Different platforms’ search engines return different kinds of results—often biased toward specific domains.

The paper reports average numbers of URLs returned per query (platform-specific search engines return relatively small sets compared to typical web search):

Then they analyze domain preferences among top-ranked returned domains. They find:
- top domains account for a large fraction of results (e.g., top-10 domains are about 21.3% of returned results for ChatGPT and 32.3% for Grok in invivo),
- preferences differ drastically across platforms.

Examples:
- reddit.com and youtube.com dominate results for ChatGPT and Grok and are also prominent for DeepSeek, but are absent in Claude’s results.
- xiaohongshu.com is prominent in DeepSeek but completely absent on other platforms.

So the search tool itself shapes the evidence the agent can access, and therefore what it can credibly support.

How Responses Get Grounded: Citations, Attribution, and the “Uncited Evidence” Problem

Now we get to the final stage: how retrieved information becomes claims in the final answer.

The paper breaks down responses into atomic claims and checks:
- whether claims are supported by cited evidence
- whether claims are supported by uncited but retrieved evidence
- whether claims are grounded at all in the observable search trace

Citation patterns differ widely

Across platforms, responses include on average between 1 and 8 citations. But citation behavior varies by platform and between invivo and invitro settings.

They also compute citation rate:

the fraction of search result URLs that eventually get cited

In invivo, citation rates range roughly from ~1.7% to ~34% depending on platform/model. In invitro, they report values like 10.6% (GPT-5.3-chat) up to around 39.7% (Claude Sonnet 4.6).

Even more telling: most cited URLs are URLs returned by Web search, but a small portion of cited URLs can be tied to “parametric citations” (generated from model internal knowledge). Across platforms, that parametric share is small—less than ~1% to 5%.

Agents cite more “authoritative” domains—mostly

The authors assess domain authority using Tranco ranks (lower rank = more authoritative/popular). Their analysis suggests:
- cited URLs (especially those from parametric domains) are often drawn from higher-authority domains than the overall set of search results,
- implying agents preferentially select certain kinds of sources for citation.

So citations can look strong. But that’s not the end of the story.

Some claims are supported by retrieved sources that are never cited

This is the core reliability concern.

They evaluate grounding in two steps:
1. Is the claim supported by evidence associated with its nearby citations?
2. If not, is it supported by other cited URLs elsewhere in the answer?
3. If not, is it supported by URLs that were returned in search but never cited?

They find:
- many grounded claims are supported by associated citations (about 30% to 46% of grounded claims, depending on setting),
- but a substantial fraction of claims are supported by other cited URLs later in the response,
- and importantly, claims can be grounded in uncited search results.

For ChatGPT, Claude, and Grok, the authors report uncited-but-supported claims account for about 14% to 53% of all claims across invivo and invitro.

That’s a big range, but the direction is the same: agents may use evidence without giving you credit to the URL. If you’re trying to audit what the assistant did, this weakens the usefulness of citations.

Ungrounded claims exist—and they score worse on factuality

After accounting for evidence tied to cited and uncited search results, some claims remain ungrounded. The paper reports ungrounded claim fractions around 15% to 20% of all claims.

When they compare factuality, grounded claims rank higher than ungrounded ones:
- claims grounded in associated citations and other cited sources are highest (example values around 3.47 and 3.35 for ChatGPT),
- claims supported only by uncited search results are slightly lower (example around 3.33 for ChatGPT),
- ungrounded claims score lowest (example around 2.84 for ChatGPT).

In plain terms: when the bot can’t point to evidence in the search trace, the answer is more likely to be unreliable.

Key Takeaways

• Search decisions vary wildly by model/platform. In a controlled test over 1,000 prompts, Claude called Web search for 83%, while GPT-5.3-chat did so for 14%.
• More Web searching doesn’t guarantee better answers. Prompts where the agent didn’t search often score higher than prompts where it did—suggesting models can correctly judge when external retrieval isn’t needed.
• Query strategy is complex and platform-specific. Agents rewrite prompts into concise queries, run multiple queries, and refine specificity over iterations; some platforms favor fan-out, others prefer deep sequential iterations.
• Search results are not neutral. Platform search engines return few results per query (roughly 5–15 URLs on average) and show strong domain preferences—sometimes excluding entire popular sources (e.g., reddit.com/youtube.com absent from Claude results).
• Citations are useful—but not always complete. A non-trivial share of claims are supported by uncited search results, raising attribution concerns, and some claims remain ungrounded (~15–20%), correlating with lower factuality.

If you design or use conversational retrieval systems, the practical lesson is blunt: evaluating “does the agent search?” is not enough. You need to evaluate the full lifecycle—when it searches, what it retrieves, how it queries, and whether it credits what it actually used.