AI Explanations That Break: Why Some Answers Confuse Users

AI Explanations That Break: Why Some Answers Confuse Users

Introduction: When “Good” Text Still Doesn’t Compute

If you’ve ever asked an AI for help, read the answer, and thought, “Wait… what am I supposed to do with this?”, you’ve experienced the real problem behind high-stakes AI use: sometimes AI responses become hard to understand in practice—even when they look fine at first glance.

New research from the original paper digs into exactly when that breakdown happens during real human–AI conversations, using data from more than 84,000 interactions across Finance and Health. The researchers focus on a practical question: when does an AI’s response push the user past the point where the conversation can smoothly continue?

What’s especially interesting is how they measure difficulty. Instead of relying only on surveys or lab tests, they look for “repair behaviors” inside the chat—like when users repeatedly re-ask the same thing, or when they explicitly ask for clarification because the prior answer didn’t make sense. That means the study isn’t just asking whether AI answers could be clearer; it asks whether they actually fit the cognitive demands of real tasks in real dialogue.

Why This Matters: The UX Problem Hiding Inside “Cognitive Load”

This research is significant right now because generative AI is no longer a novelty—it’s increasingly embedded in activities where misunderstanding can be costly. Think: “Should I buy this investment?” or “Is this symptom serious?” Or even simpler: “Compare these two options for me.” These aren’t one-shot questions; they’re multi-step thinking processes, and chat-based AI is where thinking gets disrupted.

A concrete scenario where this could be applied today: imagine a clinic or telehealth service that uses an AI assistant to explain after-visit instructions. The assistant might produce answers that are (on paper) readable and well-structured. But if the response format accidentally increases cognitive demands for certain user intents—say, “help me analyze what to do next”—patients may respond by repeating questions or requesting clarification about the AI’s response itself. In other words, the system creates friction and raises the risk of incomplete understanding.

What makes this work build on earlier AI research (and push beyond it) is the lens. Previous studies often treat “complexity” as something that scales with one variable: longer answers, denser answers, more technical language. This paper argues that’s too simplistic. The relationship between response characteristics and comprehension difficulty depends on how those characteristics combine, plus what the user is trying to accomplish. In that sense, it’s less “AI is too complex” and more “the conversational design is in the driver’s seat.”

What the Researchers Actually Measured (And Why It’s Smarter Than Surveys)

The dataset: 84,000+ real chats, split into coherent “subchats”

The researchers analyzed a large corpus of naturally occurring conversations involving ChatGPT and Gemini. The dataset draws from a previously built source collected via Measure Protocol, opt-in by US-based participants who were compensated. The study extends observations through July–October 2025, yielding 2,626 users and more than 1.1 million prompts overall.

But they don’t treat every session as one continuous interaction. Users often shift topics. So the team segmented long chat histories into topically coherent subchats (they call them “conversations” for analysis). After segmentation, they ended up with about 395,000 subchats in total, and then focused on two high-stakes domains:
- Finance: ~43,100 conversations from 1,745 users
- Health: ~41,500 conversations from 1,558 users

Repair behaviors as behavioral indicators of “conversational difficulty”

Instead of directly measuring cognitive load (which would require lab tasks or instrumentation), they use a clever proxy: what users do when things don’t land.

They focus on two kinds of “conversational repair”:

1. Fragmented repeated prompting
   This is when a user substantially repeats or reformulates an earlier request within the same conversation. The researchers implemented a rule-based method to detect repetition while filtering out acknowledgements (“thanks,” “ok”), greetings, and other non-request messages. They require prompts to meet eligibility constraints (e.g., containing at least two content tokens after normalization, not mostly non-ASCII characters, etc.).
   They compare each eligible prompt to all earlier eligible prompts in the conversation and flag it as repetition if similarity exceeds a threshold of 0.72 (chosen empirically from pilot evaluation).

2. Clarification following AI-induced misunderstanding
   This captures moments where users explicitly signal they can’t interpret the AI response and ask for clarification about that response. The team trains a DistilBERT classifier to tag user messages with labels including:

   • Task clarification
   • AI-induced misunderstanding (the one they prioritize)

   They achieve high classification performance for misunderstanding: F1_macro = 0.971 (reported for the best checkpoint on the label task).

So the outcome variables aren’t “the user felt confused.” They’re “the user tried to get the system back on track.” That’s often what you want in interface design: reliable signals of where understanding fails.

Response Complexity Isn’t a Single Slider—It’s a Budget That Depends on Context

The study’s main claim: simple features don’t predict difficulty consistently

The authors measure several linguistic and structural characteristics of the AI response, including:
- Response length (avg words per turn in the pre-event window)
- Readability using Flesch Reading Ease
- Formatting density (how many Markdown-like formatting elements appear per ~30 words)
- Lexical diversity using MTLD (Measure of Textual Lexical Diversity)
- Information density (content words vs. function words ratio)

Here’s the key twist: none of these features has a stable, universal relationship with conversational breakdown.

Longer responses are not always worse. Lexically diverse responses are not always better. Formatting is not automatically protective. The paper’s results show the relationships can even flip depending on what else is happening in the response.

The “conversational complexity budget” idea (a useful analogy)

The researchers propose a conceptual framework: a conversational complexity budget.

Think of it like packing for a trip. You have limited “capacity” for stress, attention, and working memory. If you pack only a few things (low complexity), you’re fine. But if you pack multiple heavy items together—say, a bulky suitcase and lots of fragile stuff and you’re trying to carry it up stairs—the same item that felt manageable alone can become overwhelming.

In their results, one response characteristic might be easy to process until other demands stack on top. That’s exactly what they show in the interaction modeling: difficulty is about configuration, not isolated text features.

(They link this idea back to work like Cognitive Load Theory, but they apply it to conversational AI specifically—again, see the main paper for the full framing in Wang, Bilal, Zaman.)

When Breakdown Hits: User Intent Matters More Than “Decision Authority”

Higher-order tasks cause more repair—even when AI isn’t fully “making decisions”

The study classifies user intent into categories like:
- Simple retrieval
- Learning / education
- Comparison & optimisation
- Personal financial analysis
- Complex research
…and others.

They also map intents onto a decision authority gradient (Inform → Shape → Act), but the results show something subtle: breakdown isn’t monotonic with delegated decision-making.

In both Finance and Health:
- Simple Retrieval is associated with lower odds of both repair behaviors.
- Higher-order intents (research, analysis, comparison, personalized application) are associated with higher odds.

Concrete Finance results (odds ratios, OR):
- Complex Research: more fragmented repeated prompting (e.g., OR ≈ 2.13–2.16) and more clarification after misunderstanding (OR ≈ 1.66–1.81 depending on model).
- Simple Retrieval: lower odds of repeated prompting (OR ≈ 0.81) and lower odds of clarification (OR ≈ 0.74–0.75).
- Financial Learning & Education is a standout: even though it leans toward the “Inform” end, it still shows elevated breakdown, especially clarification (OR ≈ 2.57–2.59).

Health shows a similar pattern:
- Complex Research: repeated prompting about 1.7×
- Health Data Analysis: repeated prompting about 1.6×
- Comparison & optimisation: elevated odds (around 1.44–1.46×)
- Simple Retrieval: again lower odds of repeated prompting (OR ≈ 0.84–0.85)

Health has content-driven hotspots that Finance doesn’t mirror as clearly

The “what are we talking about?” variable matters too, but differently across domains.

• Finance: after controlling for intent and other factors, substantive categories mostly don’t strongly predict breakdown—except Business Finance, which shows higher odds of both repair outcomes.
• Health: disease-focused conversations generally lead to more repeated prompting, and clarification after misunderstanding spikes in areas like psychological, identity & self-perception and social/function/environmental impact.

Practical implication: if you’re designing an AI assistant, you may need domain-specific intervention strategies, not one-size-fits-all improvements to clarity.

The Biggest Surprise: Lexical Diversity Helps—Until Responses Get Long

Main-effects models: “nothing is reliably good or bad”

In their main-effects (single-predictor) analyses, they found limited straightforward links:
- In Finance and Health, more Markdown formatting density is associated with higher odds of repeated prompting.
- Readability and information density don’t consistently show clear main effects.

But that’s not the whole story, because the paper focuses heavily on interactions (how features combine).

Interaction #1 (cross-domain): Lexical diversity × response length flips

This is the clearest replicated result across Finance and Health.

They find:
- When AI responses are short, greater lexical diversity (higher MTLD) is associated with less fragmented repeated prompting.
- As responses get longer, that advantage weakens.
- Around ~600 words (a model-implied crossover point specific to the observed data, not a universal threshold), the relationship can even reverse direction.

That suggests a counterintuitive design lesson: varied vocabulary can make short explanations feel richer and clearer—but if you scale it up, lexical variety might start acting like friction, especially during tasks that require integration and comparison.

Interaction #2 (Finance): Length interacts with both information density and readability

For clarification after misunderstanding in Finance, they find that the effect of response length depends on other characteristics:

1. Information density × response length

   • When information density is low, longer responses are linked to lower predicted misunderstanding.
   • When information density is high, longer responses are linked to higher predicted misunderstanding.
     Their model-implied crossover is around 150–200 words (again: data-specific).

2. Readability (Flesch) × response length
   Even readability isn’t stable across length. The paper reports:

   • For responses with lower reading ease (harder-to-read), predicted misunderstanding decreases with length.
   • For responses with high reading ease (easier-to-read), predicted misunderstanding increases with length—most strongly among the longest responses.

This is a great example of why “surface simplicity” is not the same as comprehension. A response can be easy to read word-by-word but still demand heavy mental processing because of how much it tries to hold together.

Interaction #3: Readability depends on the user’s prompt style

In Finance, the authors also observe that AI readability interacts with user-prompt readability. Easier-to-read AI outputs are associated with more misunderstanding when the user prompt is themselves harder to read—suggesting a fit problem between user input style and response style.

This aligns with the older idea that prior knowledge and expertise change how people process information, but here it’s captured behaviorally in chat: what works for one conversational pattern may not work for another.

Model Provider Effects: Gemini Shows More Repair Than ChatGPT

The paper also finds an association between which AI system was used and conversational breakdown—even after controlling for intent, topics, user prompts, and measured response text properties.

They report (Finance):
- Gemini has about 4× the odds of fragmented repeated prompting compared to ChatGPT (OR ≈ 4.06 in main-effects, OR ≈ 4.03 in interaction model).
- Gemini has about 2.5–3× the odds of clarification after misunderstanding (OR ≈ 2.80 main, OR ≈ 2.99 interaction).

In Health, Gemini also shows higher breakdown:
- Repeated prompting: about 3.7–3.9×
- Clarification: about 2.5×

Important caveat: since users weren’t randomly assigned to models, these aren’t causal comparisons. The paper explicitly warns that unmeasured response characteristics—like coherence, explanatory sequencing, contextual alignment, or reasoning structure—could be driving the residual differences.

Still, from a product-design perspective, it’s a strong signal that “model choice” isn’t invisible when it comes to user comprehension friction.

What This Means for Designers Building Real AI Assistants

Don’t optimize “simplicity” as a single objective

A common interface intuition is: “Make responses shorter. Use simpler words. Add structure.” This paper complicates that.

Because relationships flip, the better question becomes:

Does this response configuration fit the user’s intent and the conversation’s current cognitive load budget?

For example:
- Lexical diversity might be beneficial early or in shorter responses.
- Formatting might increase repair when it adds complexity without clarifying the next step.
- Longer responses might be helpful only when information density stays manageable.

Use conversational repair as a live signal for adaptation

The repair behaviors—repetition and clarification—are not just research proxies. They can become real-time instrumentation in production:
- If a user starts repeating prompts, the system should treat that as a signal the current explanation strategy isn’t working.
- If the user asks clarification after misunderstanding, the system should shift the response configuration: break down, reduce density, change explanation level, or ask the user what part is unclear.

The paper doesn’t claim which adaptation strategy wins, because the data is observational—but it strongly motivates testable hypotheses.

Try intent-aware explanation strategies

Since breakdown varies by intent, an assistant shouldn’t respond the same way to every question. A workflow that makes sense:
- For simple retrieval, provide targeted answers.
- For learning, provide scaffolding and check comprehension.
- For comparison/optimisation and analysis, explicitly manage cognitive load by controlling how much is introduced at once.

The evidence suggests that comprehension issues rise not just with “decision delegation,” but with integration-heavy goals.

Design for context fit: user prompt style and task framing matter

The readability interaction suggests that the system should consider how the user framed the question. Even without “knowing” user skill, the assistant can infer from prompt characteristics whether a given response style is likely to fit.

That’s the “relational comprehensibility” direction the paper pushes: comprehension is not a property of the response alone.

Key Takeaways

• Conversational difficulty isn’t reliably predicted by a single text feature like length, readability, or lexical diversity on its own.
• The study uses behavioral repair signals—not lab measures—as indicators of breakdown:
  • Fragmented repeated prompting
  • Clarification after AI-induced misunderstanding
• User intent matters a lot. Higher-order tasks (research, analysis, comparison, personalized application) show elevated breakdown; simple retrieval shows lower breakdown.
• The clearest cross-domain interaction: lexical diversity helps with repeated prompting only when responses are relatively short, and the benefit weakens as responses get longer.
• In Finance, response length’s effect depends on what’s packaged inside:
  • Length + information density can either reduce or increase misunderstanding.
  • Length + reading ease can also flip.
• The authors propose a “conversational complexity budget”: cognitive demands combine, so simplicity isn’t a one-dimensional goal.
• Model provider correlates with breakdown (Gemini shows higher odds of both repair behaviors), though the study can’t prove causality.
• For real products: treat repair behaviors as live signals to adapt response configuration to the user’s goal and the conversation’s current load—rather than trying to globally “make answers simpler.”

If you want, I can also turn these findings into a practical checklist for “response configuration” (what to adjust based on intent, response length, and density) that teams can test A/B in a deployed assistant.