Reasoning

Self-Consistency

Self-consistency is a technique where you ask a model the same question several times with sampling turned on, collect the independent reasoning paths, and take the most common final answer. It trades extra compute for accuracy on problems that have a single checkable answer.

Best for
Questions with one checkable answer where accuracy beats cost
Difficulty
Intermediate
Updated

What is Self-Consistency?

The intuition: a hard problem usually has many wrong ways to reason but the right answers converge. If five out of seven independent attempts land on 42 and the other two each land somewhere different, 42 is a much better bet than any single attempt.

It is also a free confidence signal. Seven out of seven agreeing is very different from three out of seven, and you can route low-agreement cases to a human or a stronger model. For free-form outputs that cannot be voted on directly (summaries, code), a later variant called Universal Self-Consistency asks the model itself to pick the most consistent candidate.

When to use it — and when not to

Use it when

  • Maths, logic, extraction or classification where answers can be normalised and compared exactly.
  • Batch jobs where a wrong answer costs more than a few extra calls.
  • When you want an agreement score to decide what needs human review.

Skip it when

  • Open-ended writing — there is nothing to vote on without a judge step.
  • Latency-sensitive chat: N samples means N times the cost and, unless run in parallel, the wait.
  • Errors that are systematic. If the model reliably misreads the question, all samples agree on the wrong answer.

Copyable template

Sample prompt plus aggregation step 
SAMPLE PROMPT — run this [N, e.g. 5–9] times as separate calls, temperature around 0.7–1.0:

[QUESTION]
Reason step by step, then give the final answer on the last line exactly as:
FINAL: [ANSWER IN A FIXED FORMAT, e.g. a number with no units / one label from the list]

AGGREGATION — in code (preferred), or as one last call:

Here are [N] independent answers to the same question:
[PASTE THE FINAL: LINES]
Normalise equivalent answers, then report the majority answer, how many of the [N] agreed, and the word LOW-CONFIDENCE if fewer than [THRESHOLD, e.g. 60%] agree.

Majority voting is a few lines of code once every sample ends with a fixed “FINAL:” line. Use an odd N to avoid ties.

Replace everything in [SQUARE BRACKETS] before you run it.

Worked example: Extracting a contract notice period

A legal-ops team extracts the termination notice period from hundreds of contracts, where clauses are often written awkwardly.

Prompt 
Read the clause and state the notice period the customer must give to terminate.
Clause: "Either party may terminate this Agreement at the end of the Initial Term by giving not less than ninety (90) days' written notice, provided that where the Customer has elected quarterly billing the notice period shall be one calendar quarter."
The customer is on annual billing.
Reason step by step, then give the final answer on the last line as: FINAL: [number] days
Illustrative outputWritten to show the pattern, not a recorded model run
7 samples returned:
FINAL: 90 days   ×6
FINAL: 91 days   ×1   (counted a calendar quarter)

Majority: 90 days (6/7 agree, 86%). Not flagged.

Why it works: One unlucky sample took the quarterly branch. Voting absorbs it, and the agreement score shows the case is safe to accept without review.

Common mistakes

  1. Temperature 0. Deterministic sampling gives you the same path N times, so the vote means nothing. Sampling must be on.
  2. Asking for N answers in one call. “Give me five solutions and pick the most common” produces correlated attempts that copy each other. Use separate calls.
  3. Answers you can’t compare. “About ninety days”, “90” and “three months” are the same answer. Fix the format, then normalise before counting.
  4. Ignoring the agreement score. The vote margin is the most useful by-product. Route low-agreement items to review rather than trusting the majority blindly.
  5. Using it where one good call suffices. On easy questions all samples agree and you have paid N times for nothing. Spot-check before running it at scale.

Origin & sources

Proposed by Xuezhi Wang, Jason Wei, Denny Zhou and colleagues at Google in 2022 (ICLR 2023) as a replacement for greedy decoding in chain-of-thought prompting. They reported large gains over standard CoT, including +17.9% on the GSM8K maths benchmark. Chen et al. (2023) extended the idea to free-form outputs with Universal Self-Consistency.

Try it on The Prompt Index

AI Labs Run the same prompt across many models and compare their answers side by side. Prompt Optimizer Tighten the answer format so samples can be compared.

Frequently asked questions

How many samples does self-consistency need?

Gains are often visible from five samples and flatten out as you add more. An odd number between five and nine is a practical starting point; measure on your own data before going higher.

What temperature should I use for self-consistency?

High enough that samples genuinely differ, typically around 0.7 to 1.0. At temperature 0 every sample is the same and the vote is meaningless.

Can self-consistency work for writing or code?

Not with a simple majority vote, because no two outputs are identical. Universal Self-Consistency asks the model to choose the most consistent candidate, and for code you can vote on which outputs pass the same tests.