Rude vs Neutral Prompts Can Change LLM Cost by 44%

Rude vs Neutral Prompts Can Change LLM Cost by 44%

Introduction

If you’ve ever noticed that an LLM “acts” differently when you phrase the same question more politely—or more aggressively—you’re not imagining things. New research from this arXiv paper goes one step further: it shows that prompt tone can change both answer accuracy and the actual inference cost of modern LLMs, measured through how many output tokens the model generates.

That matters because companies usually pay for model usage based on tokens, especially the billable output tokens. So even if the “right answer” rate stays similar, a tone that makes the model ramble more can quietly inflate cost. The paper tests this directly by running the same multiple-choice questions under seven tone conditions, ranging from sycophantic to threatening, and tracks both accuracy and output-token consumption.

The experiments use a 570-question subset of MMLU (10 questions from each of 57 subjects), and they evaluate four recent models: ChatGPT-4o, ChatGPT-5-nano, Gemini 2.5 Flash, and Gemini 2.5 Flash Lite. Across these models, the authors find that output-token length variation dwarfs accuracy variation—and the token swing can be huge.

Why This Matters (Beyond “Politeness”)

Prompt tone is often treated like a user-experience detail—like whether the assistant says “please” or sounds annoyed. But this research highlights something more operational: tone is effectively a control knob for cost.

Why this is especially relevant right now: AI products are being deployed at scale, and token-based pricing turns tiny behavioral shifts into real money. The paper reports output-token consumption changing by up to 44.3% across tone conditions (most dramatically for Gemini 2.5 Flash Lite). That’s not a rounding error—it’s the kind of difference that can change monthly inference bills, rate limits, and even latency.

Here’s a scenario you could apply today: imagine a customer-support bot answering the same troubleshooting question. Some users naturally communicate with neutral language (“Please help…”). Others come in frustrated (“Just fix it!”). If the model generates significantly longer reasoning traces or explanations depending on tone, you might see unpredictable cost and slower responses—even when the question content is identical. This paper suggests you can’t safely assume that “same prompt, same meaning” implies “same token usage.”

And it builds on previous AI research in a consistent direction: earlier work showed prompt phrasing can change accuracy a lot (sometimes rankings across prompt templates correlate only weakly). This study adds a new dimension—not just quality sensitivity, but cost sensitivity—and it connects that to the real-world “metering” of LLM services.

What the Researchers Actually Measured (Accuracy vs Output Tokens)

The core idea is straightforward: LLMs generate text token by token, and output tokens are the part most services meter as billable inference. So the authors use average generated output token count as a proxy for inference cost.

Experimental setup in plain language

• Dataset: 570 MMLU multiple-choice questions (4 options, labeled A/B/C/D)
• Tone conditions: 7 prefixes, from sycophantic to threatening
• Runs: 10 repeated runs per model + tone
• Determinism controls:
  • For ChatGPT-4o, Gemini 2.5 Flash, and Gemini 2.5 Flash Lite: temperature = 0
  • For ChatGPT-5-nano: temperature wasn’t adjustable, default used
  • For Gemini “thinking budget”: set to 0 to avoid hidden reasoning tokens and keep token reporting aligned with visible output

Important: the authors kept the tone prefix length roughly similar (so token differences aren’t just “one prompt is longer than another”). They also used a lightweight sentiment check with VADER compound scores to verify the tone continuum, reporting a trend from about 0.95 (sycophantic) to about -0.77 (threatening).

The tradeoff signal

Across all four models, the standout result is:

• Output token length varies far more than accuracy
• The authors explicitly show output-token swings up to 44.3% within the same model across tones
• For accuracy, the range is comparatively small (e.g., less than 1.5% for the two ChatGPT models)

Below is the paper’s summary of mean accuracy and output token length (mean ± range across tones):

Key numerical highlights from the paper:
- Output-token ranges across worst vs best tone:
- ChatGPT-4o: 67.4 tokens (≈ 23.2%)
- ChatGPT-5-nano: 180.4 tokens (≈ 13.1%)
- Gemini 2.5 Flash: 217.1 tokens (≈ 23.0%)
- Gemini 2.5 Flash Lite: 973.5 tokens (≈ 44.3%)

How Different Models React to Tone (And Why “Best” Changes)

A big lesson from this study: there isn’t one universal “tone that’s best.” The optimal tone depends on the model.

The paper’s main behavioral pattern (in human terms)

The authors propose that tone shifts the model’s response policy—things like verbosity, caution, and the amount of “checking work” it performs before concluding.

Think of tone like a steering wheel that nudges the car’s driving style. You still reach the destination (answer letter), but the route (number of output tokens) can be very different.

Model-by-model takeaways

Here’s how the paper interprets the accuracy-vs-token “frontier” for each model:

One particularly striking point:
- The paper says Gemini 2.5 Flash Lite showed no run-to-run variation in either accuracy or output tokens under identical prompts. In other words, it’s highly consistent for this setup.

Meanwhile, token sensitivity varies a lot:
- ChatGPT-4o is relatively stable (smallest output-token range in absolute terms)
- Gemini 2.5 Flash Lite is the most tone-sensitive in cost terms (44.3% token swing)

The Accuracy–Cost Frontier: It’s Not Just “More Tokens = Better”

A common intuition is: “If the model writes more, it must be thinking more, so accuracy should improve.” This study complicates that.

Why token count isn’t a pure proxy for “quality”

The authors explain that output tokens can represent both:
- useful extra reasoning/checking, and/or
- wasted verbosity (“overthinking” beyond the necessary depth)

So the relationship between output tokens and correctness can be non-monotonic depending on architecture and prompting.

Pareto-optimal “efficient tones”

The paper defines a tone as Pareto-dominated if another tone has higher or equal accuracy and lower or equal token length (with at least one strict improvement). They report which tones sit on the efficient set:

So sometimes the best “accuracy-per-token” isn’t the most polite and certainly isn’t the most extreme. It’s about how the model’s reasoning and verification steps get triggered.

A practical “efficiency per output tokens” metric

They introduce a metric: correct answers per 100,000 output tokens.

Most token-efficient tone reported:

That’s a really practical framing for developers: not just “accuracy,” but “accuracy relative to generated text volume.”

What This Suggests for Real Systems (And What You Can Do)

Now for the part you can actually use when building or evaluating products.

1) Don’t treat tone as cosmetic in evaluation

Many benchmarking pipelines assume prompts are semantically equivalent if the question and answer options stay the same. But here, tone changed output-token usage by up to 44.3% even with similar prefix lengths.

So evaluation setups should either:
- canonicalize tone (use a single standardized style), or
- include tone as a controlled variable, reporting a distribution rather than a single point estimate.

This is consistent with the broader push in the paper toward evaluation protocols that account for prompt sensitivity (the authors cite earlier work that shows accuracy can swing dramatically with tiny prompt changes).

2) Add “cost metrics” to your acceptance criteria

If you’re tracking only accuracy, you’ll miss the hidden cost regressions. This paper uses output token count as a proxy for inference cost, which is exactly what you’d want for token-metered APIs.

A good production-minded metric set could include:
- accuracy (or task success rate)
- output tokens (completion tokens)
- parse success rate (since errors can trigger reruns or fallbacks)
- end-to-end latency (often correlated with output length)

3) Be careful with system instructions that request verbose reasoning

The authors intentionally used a system instruction that asked for step-by-step reasoning and a final CONCLUSION: X format. In real products, you might not show full reasoning to users—but the internal generation still affects output tokens unless hidden.

Their results strongly imply that the amount of reasoning trace the model emits can be tone-dependent. If you’re trying to control cost, you may need:
- tone normalization,
- shorter response templates,
- or reasoning-control strategies that reduce unnecessary output.

4) Choose “sweet spot” tones per model (but don’t use this as permission to be rude)

One of the more delicate conclusions: the paper finds that extreme tones are not consistently desirable for overall performance. For the models tested, the best trade-off tends to land on the rude/neutral boundary depending on the architecture—but the ethical conclusion is clear: they do not advocate rude behavior in human conversation.

So from a product perspective, the safe approach is: standardize the tone for the model, not the users. You can interpret the user’s tone without passing it through in full.

Key Takeaways

• Tone affects both answer quality and inference cost. The study measures inference cost via output token count and finds large differences even when the question content is unchanged.
• Output tokens vary more than accuracy. Across models, token-length variation substantially exceeded accuracy variation, with output tokens changing by up to 44.3% across tone conditions (worst for Gemini 2.5 Flash Lite).
• The “best” tone is model-dependent.
  • ChatGPT-4o: rude tone dominated (better accuracy + fewer tokens)
  • Gemini 2.5 Flash: neutral was best for accuracy, while rude was cheapest (clear tradeoff)
  • Gemini 2.5 Flash Lite: neutral dominated and behavior was extremely consistent run-to-run
• More tokens aren’t guaranteed to mean higher accuracy. Token count can reflect either useful checking or wasted verbosity, depending on the model and tone.
• For production: treat tone as an operational variable. Canonicalize prompts or include tone conditions in evaluation so you don’t get surprised by cost/latency swings.
• Don’t copy the rudeness—copy the control. Standardize tone in your model interface to manage cost reliably, while keeping user-facing behavior respectful.

If you want, I can also turn these findings into a practical evaluation checklist (what to log, what to standardize, and how to pick a “cost-aware” prompt template) tailored for your specific use case.