Self-Consistency and Answer Voting

By Michael Samuel Naeem · March 15, 2025 · 5 min read

The support bot answered "Tuesday" to a shipping ETA question on the first try. On the second identical request — same model, temperature 0.7 — it said "Wednesday." Same tracking data, same prompt template, different stochastic path through the decoder. Self-consistency and answer voting exploit exactly this property: independent samples that agree are more likely correct; samples that disagree signal uncertainty you should surface to a human instead of picking one at random.

How self-consistency works

Standard chain-of-thought asks the model to reason step by step, then give a final answer. Self-consistency runs that process N times:

Prompt ──→ Sample 1 → reasoning → Answer A
       ──→ Sample 2 → reasoning → Answer A
       ──→ Sample 3 → reasoning → Answer B
       ──→ Sample 4 → reasoning → Answer A
       ──→ Sample 5 → reasoning → Answer A

Majority vote → Answer A (4/5)

Errors in individual chains often diverge; correct reasoning converges on the same final result for well-posed problems.

Implementation pattern

import asyncio
from collections import Counter

async def sample_one(client, prompt: str, temperature: float = 0.7) -> str:
    resp = await client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
    )
    return resp.choices[0].message.content

def extract_answer(full_response: str) -> str:
    # Task-specific: parse "Final answer: X" or last JSON field
    if "Final answer:" in full_response:
        return full_response.split("Final answer:")[-1].strip().lower()
    return full_response.strip().lower()

async def self_consistent_answer(prompt: str, n: int = 5) -> tuple[str, float]:
    responses = await asyncio.gather(*[sample_one(client, prompt) for _ in range(n)])
    answers = [extract_answer(r) for r in responses]
    counts = Counter(answers)
    winner, vote_count = counts.most_common(1)[0]
    confidence = vote_count / n
    return winner, confidence

Run samples in parallel to bound latency to roughly one inference round-trip plus overhead.

Prompt design for extractable votes

Self-consistency requires normalizable answers:

Analyze the order status and determine the delivery day.

Think step by step in <reasoning> tags.
Then output exactly one line: Final answer: <weekday name>

For JSON outputs, vote on the parsed field:

def extract_json_answer(text: str) -> str:
    data = json.loads(extract_json_block(text))
    return str(data["category"]).lower()

Ambiguous extraction — "the answer is probably Tuesday" — breaks voting. Enforce strict output format.

Weighted and semantic voting

Plain majority assumes all samples equal. Improvements:

Confidence weighting — if the model returns logprobs, weight votes by average token probability of the answer span.

Semantic clustering — embed free-text answers, cluster by similarity, vote on cluster centroids:

def semantic_vote(answers: list[str], embed_fn) -> str:
    vectors = [embed_fn(a) for a in answers]
    # cluster with threshold cosine similarity
    clusters = cluster_by_similarity(answers, vectors, threshold=0.85)
    largest = max(clusters, key=len)
    return Counter(largest).most_common(1)[0][0]

Useful when phrasing varies but meaning matches ("Tue" vs "Tuesday").

When to escalate on low confidence

answer, confidence = await self_consistent_answer(prompt, n=7)

if confidence < 0.6:  # no majority
    return escalate_to_human(reason="low_vote_confidence", samples=answers)

if confidence < 0.85 and is_high_stakes(context):
    return escalate_to_human(reason="narrow_margin", answer=answer)

Low agreement is a feature — it tells you the model is uncertain even when each individual response sounded confident.

Cost and latency tradeoffs

Mitigations:

• Use smaller model for samples, larger for tie-breaker only
• Adaptive sampling — stop early if 3/3 agree
• Cache identical prompts within session

async def adaptive_vote(prompt: str, max_n: int = 9) -> str:
    answers = []
    for i in range(max_n):
        answers.append(extract_answer(await sample_one(client, prompt)))
        counts = Counter(answers)
        top, top_count = counts.most_common(1)[0]
        if top_count >= 3 and top_count / len(answers) >= 0.67:
            return top
    return counts.most_common(1)[0][0]

Comparison to other techniques

Self-consistency is the sweet spot when you lack a trained reward model but need better reasoning than greedy decode.

Evaluation

Measure on your task set:

• Accuracy with vs without voting
• Vote margin distribution
• Cost per correct answer

If majority vote rarely beats single sample, your task may not have convergent reasoning paths — do not pay 5× indefinitely.

Treat production rollout as a measured change: ship with observability, validate rollback, and review metrics 24 hours after deploy — patterns that look obvious in docs fail when skipped under release pressure.

Common production mistakes

Teams get self consistency voting wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

LLM features around self consistency voting break in production when prompts assume deterministic output, context windows are sized for dev datasets, or token costs are never budgeted per user session. Always log prompt hash, model version, and latency—not raw prompts with PII.

Debugging and triage workflow

When self consistency voting misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• Self-Consistency Improves Chain of Thought Reasoning (Wang et al.)
• OpenAI API parallel requests
• LangChain self-consistency chain
• Tree of Thoughts paper
• Semantic similarity with sentence-transformers