Cutting LLM Costs: Caching, Routing, and Batching

By Michael Samuel Naeem · February 3, 2026 · 6 min read

Most teams discover their LLM bill the same way I did: a Slack message from finance asking why the "AI feature" costs more than the database cluster. The good news is that LLM spend is one of the most compressible line items in a modern stack. Three levers — caching, routing, and batching — routinely take 40-80% off a bill without users noticing anything except, sometimes, faster responses.

LLM cost optimization is not one trick; it's a stack of them applied in the right order. Start with caching because it's nearly free to enable, then add routing to stop overpaying for easy work, then batch everything that isn't user-facing. Here's how each works and what to watch for.

Start with the token math

Every optimization decision comes back to one invoice: you pay per input token and (more, usually 3-5x more) per output token. Before touching code, pull a week of traffic and bucket it. I usually find the distribution is lopsided — a handful of endpoints generate most of the spend, and it's almost always input tokens dominating because of fat system prompts and retrieved context.

A quick audit table for one service I worked on looked like this:

The /summarize-doc and /chat rows are where the money is, and both are cache- and batch-friendly. Do the audit; intuition about where the cost lives is usually wrong.

Prompt caching: the cheapest win

If your requests repeat a large fixed prefix — a system prompt, tool definitions, a policy document, few-shot examples — you're paying to re-process identical tokens on every call. Prompt caching stores the model's internal state for that prefix so subsequent calls skip it.

With Anthropic's API you mark the stable section with a cache breakpoint:

import anthropic

client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT_AND_POLICY,  # stable, reused
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": user_turn}],  # varies per call
)

OpenAI's prompt caching is automatic for prefixes over ~1024 tokens — you just need to keep the shared content at the front of the prompt and the variable content at the end. That ordering detail matters: caching keys off the prefix, so a single changed token near the top invalidates everything after it. Put timestamps, request IDs, and user input last.

Cached input tokens typically bill at 10-25% of the normal rate. For the /chat endpoint above, moving the 3,000-token system prompt behind a cache breakpoint cut its cost by roughly 55% on its own. The cache has a short TTL (minutes), so it helps most on bursty, conversational traffic.

Semantic caching for repeated questions

Prefix caching handles identical prefixes; it does nothing for two users asking the same question in different words. That's where a semantic cache helps: embed the incoming query, look for a near-duplicate in a vector store, and return the stored answer if similarity clears a threshold.

The trap is the threshold. Set it too loose and you serve "close enough" answers that are subtly wrong; too tight and you never hit. I gate semantic cache hits behind a similarity floor (cosine > 0.93 in one FAQ bot) and only cache responses for read-only, non-personalized queries. Anything that depends on the specific user or live data never touches the cache.

Routing: stop sending easy work to the expensive model

The single biggest structural saving is model routing — matching each request to the cheapest model that can handle it. Frontier models are wonderful and wasteful; a lot of production traffic is classification, extraction, or short factual answers that a small model does just as well for a tenth of the price.

A pragmatic router has two tiers plus an escape hatch:

def route(request) -> str:
    if request.task in ("classify", "extract", "tag"):
        return "gpt-4o-mini"          # cheap tier for structured/simple work
    if request.token_estimate > 30_000 or request.needs_reasoning:
        return "claude-sonnet-4-5"     # capable tier for hard/long work
    return "gpt-4o-mini"

def answer(request):
    model = route(request)
    result = call(model, request)
    if model != "claude-sonnet-4-5" and low_confidence(result):
        result = call("claude-sonnet-4-5", request)  # escalate on failure
    return result

The escalation path is what makes this safe. Route optimistically to the cheap model, detect low-quality or low-confidence outputs (short answers, refusals, failed schema validation), and retry on the strong model only then. In practice the escape hatch fires on 10-20% of traffic, so you keep near-frontier quality while paying cheap-tier prices for most calls. Validate the whole thing against an eval set — see measuring agent quality with evals — so a routing change doesn't quietly regress accuracy.

Batching: half price for anything that can wait

Real-time chat needs low latency, but a huge amount of LLM work is offline: enriching a catalog, summarizing yesterday's tickets, generating embeddings for a backfill, running an eval suite. For those, provider batch APIs cost about half of synchronous calls in exchange for a delivery window (often up to 24 hours).

# OpenAI Batch API: submit a JSONL of requests, poll for completion
batch = client.batches.create(
    input_file_id=uploaded_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

Two rules keep batch jobs sane: make each request idempotent with a stable custom_id so you can retry without duplicating work, and design the pipeline to tolerate partial failures — a 50k-row batch will have a few rows that error, and you don't want to reprocess all 50k to recover them.

If you also control your own inference (self-hosted vLLM or similar), continuous batching at the serving layer packs concurrent requests onto the GPU and can lift throughput several times over naive one-at-a-time serving. That's a different mechanism than the provider batch API but the same principle: amortize fixed cost across more work.

Put them together, then measure

The order matters. Enable prefix caching (near-zero risk), add a semantic cache for read-only queries, introduce two-tier routing with an escalation fallback, and push all non-interactive work to batch. Layered, these compound: caching shrinks the input tokens, routing shrinks the per-token price, and batching halves whatever's left of the offline load.

Then instrument it. Track cost per request, cache hit rate, escalation rate, and — critically — quality on a fixed eval set, so you can prove a cost change didn't cost you accuracy. Cheaper output that's wrong isn't a saving; it's a refund waiting to happen. If you want a broader tour of the systems side of running models in production, my writeup on architecting real-time platforms covers adjacent territory.

Resources

• OpenAI — Prompt caching
• OpenAI — Batch API
• Anthropic — Prompt caching
• Anthropic — Message Batches API
• vLLM documentation (continuous batching)
• Google Cloud — Vertex AI batch predictions