Chain-of-Thought: Costs and Benefits

By Michael Samuel Naeem · October 31, 2024 · 5 min read

"Let's think step by step" is the most copied sentence in prompt engineering. It's also responsible for a measurable chunk of LLM spend at companies that added it to every prompt whether the task needed reasoning or not. Chain-of-thought (CoT) works — on problems that require multi-step deduction. On "What's our return window?" it generates three paragraphs of reasoning about what a return window might mean before saying "30 days."

What CoT actually does

CoT prompts the model to produce intermediate reasoning before the final answer. This helps because:

• Errors in step 3 are visible (sometimes correctable)
• Complex problems get decomposed implicitly
• Larger models leverage more compute per query (thinking tokens ≈ inference-time compute)

It doesn't help when:

• The answer is directly retrieved from context
• The task is pattern matching ("classify this email")
• The model lacks domain knowledge (reasoning can't invent facts)

Visible vs hidden reasoning

Visible CoT — user sees the steps:

Prompt: Solve step by step, then give final answer on last line.
Output:
Step 1: Calculate monthly interest...
Step 2: Apply compound formula...
Final answer: $1,847.32

Hidden CoT — model reasons, you extract the answer:

Prompt: Reason internally. Output ONLY valid JSON: {"answer": number}

Or use model features designed for this — OpenAI o-series and similar models separate reasoning tokens from output tokens.

For production UIs, hidden CoT is almost always correct. Support agents don't need the model's arithmetic narrated.

Cost math

Measure accuracy delta on your eval set. Industry benchmarks on MATH don't predict your invoice parsing task.

Selective CoT with routing

Don't apply CoT globally — route to it:

COMPLEXITY_PROMPT = """Rate query complexity 1-5.
1=trivial FAQ, 5=multi-step reasoning required.
Return JSON: {"score": int}"""

async def complete_with_optional_cot(query: str) -> str:
    score = await classify_complexity(query)
    if score >= 4:
        return await llm.complete(query, system=COT_SYSTEM_PROMPT)
    return await llm.complete(query, system=DIRECT_SYSTEM_PROMPT)

Cheaper than CoT everywhere; more accurate than CoT nowhere.

Self-consistency (CoT++)

Run CoT N times at temperature > 0, majority-vote the final answer. Accuracy goes up; cost goes up N×. Use for:

• High-stakes single decisions (medical triage suggestions, legal classification)
• Offline eval and labeling
• Low-volume premium tier

Don't use for high-QPS endpoints unless N=3 and only on the routed complex subset.

Structured reasoning formats

Free-form CoT is hard to parse. Structured steps help:

RESPONSE_SCHEMA = {
    "steps": [{"description": str, "result": str}],
    "final_answer": str,
    "confidence": float,
}

Structured output lets you log reasoning for debugging without showing it to users, and truncate if steps exceed budget.

Reasoning budget caps

Set max tokens on reasoning:

response = await llm.complete(
    messages=messages,
    max_tokens=500,          # hard cap
    stop=["Final answer:"],  # early termination
)

Monitor average reasoning length. Models drift toward verbose over time as prompts accumulate examples with long reasoning chains.

CoT with RAG

CoT + RAG has a specific failure mode: the model reasons confidently about facts not in retrieved context. Mitigate:

• Require citations per reasoning step
• Add instruction: "If context doesn't contain needed info, say so in step 1"
• Evaluate faithfulness separately from accuracy

A correct-looking chain of reasoning built on hallucinated premises is worse than a short wrong answer — it's convincing.

When CoT helps vs hurts

def should_use_cot(task_type: str, complexity: str) -> bool:
    if task_type in ("math", "logic", "multi_step_reasoning"):
        return True
    if task_type in ("classification", "extraction", "creative"):
        return False
    return complexity == "high"

CoT adds 2–4× output tokens. Using it on simple tasks wastes budget without quality gain.

Self-consistency for high-stakes decisions

Generate multiple reasoning paths, take majority vote:

async def self_consistent_answer(question: str, n: int = 5) -> str:
    paths = await asyncio.gather(*[
        llm.generate(f"{question}\nLet's think step by step.")
        for _ in range(n)
    ])
    final_answers = [extract_final_answer(p) for p in paths]
    return Counter(final_answers).most_common(1)[0][0]

5× cost, significant accuracy improvement on math/reasoning benchmarks. Reserve for high-stakes paths where wrong answer is costly — not every request.

CoT with reasoning models (o1, Claude extended thinking)

Reasoning models internalize CoT — external prompting changes:

# Standard model: explicit CoT prompt
prompt = "Solve step by step: What is 15% of 847?"

# Reasoning model: just ask the question
prompt = "What is 15% of 847?"
# Model generates internal reasoning chain automatically
# Access via response.reasoning_content (if API exposes it)

Don't add "think step by step" to reasoning models — redundant and may interfere with internal reasoning. Monitor reasoning token usage separately from output tokens.

Failure modes

• CoT on simple tasks — 3× token cost, no quality improvement
• CoT with RAG without citation requirement — confident reasoning about facts not in context
• Self-consistency on every request — 5× cost unsustainable at scale
• CoT output shown to users — verbose, confusing; keep internal for debugging
• Reasoning model + explicit CoT prompt — redundant; may degrade internal reasoning

Production checklist

• CoT used only for multi-step math, logic, and complex reasoning tasks
• Reasoning kept internal — not shown to end users unless explicitly requested
• Self-consistency reserved for high-stakes decisions (5 paths, majority vote)
• Reasoning models: no explicit CoT prompting; monitor reasoning token usage
• RAG + CoT: citation required per reasoning step
• Average reasoning length monitored — models drift toward verbosity over time

Resources

• Chain-of-Thought Prompting paper (Wei et al.)
• Self-Consistency Improves Chain of Thought
• OpenAI o1 reasoning models documentation
• Anthropic extended thinking guide
• DSPy programmatic prompt optimization