Token Budgeting for Production LLM Apps

By Michael Samuel Naeem · April 6, 2025 · 5 min read

Your LLM bill climbed from $200 to $4,000 per month without a traffic spike. The culprit: an agent loop that appends full tool results to context, a system prompt that grew to 2,500 tokens over six sprints, and no max_tokens limit on user-facing completions. Users ask vague questions and get 2,000-token answers they never read.

Token budgeting is how you keep LLM applications predictable in cost, latency, and quality. It is not about being cheap — it is about allocating a finite context window deliberately instead of letting every component consume as much as it wants.

Anatomy of a token budget

For a 128K context model serving a RAG chatbot:

Total context:     128,000 tokens
├── Output reserve:  4,000 tokens (max response)
├── Safety margin:   4,000 tokens (10% buffer)
└── Input budget:  120,000 tokens
    ├── System prompt:       2,000 tokens (fixed)
    ├── Tool definitions:    1,500 tokens (fixed)
    ├── RAG context:        8,000 tokens (variable)
    ├── Conversation history: 6,000 tokens (variable)
    └── User message:        500 tokens (variable)

Every component competes for the same window. When conversation history grows, RAG context must shrink — or you summarize history to make room.

Enforcing budgets in code

Track token counts before sending requests:

import tiktoken

encoder = tiktoken.encoding_for_model("gpt-4o")

class TokenBudget:
    def __init__(self, max_context: int, max_output: int, margin_pct: float = 0.10):
        self.max_input = int((max_context - max_output) * (1 - margin_pct))
        self.max_output = max_output
        self.used = 0

    def count(self, text: str) -> int:
        return len(encoder.encode(text))

    def allocate(self, text: str, label: str) -> str:
        tokens = self.count(text)
        remaining = self.max_input - self.used
        if tokens <= remaining:
            self.used += tokens
            return text
        # Truncate to fit
        encoded = encoder.encode(text)[:remaining]
        self.used += remaining
        return encoder.decode(encoded)

Use this in your request builder:

budget = TokenBudget(max_context=128000, max_output=4096)

system = budget.allocate(SYSTEM_PROMPT, "system")
context = budget.allocate(retrieve_context(query, max_tokens=8000), "rag")
history = budget.allocate(truncate_history(messages, budget.remaining()), "history")
user_msg = budget.allocate(user_input, "user")

Compressing conversation history

Unbounded history is the most common budget overflow. Strategies in order of preference:

Sliding window: keep the last N messages.

def sliding_window(messages: list, max_tokens: int) -> list:
    kept, total = [], 0
    for msg in reversed(messages):
        tokens = count_tokens(msg["content"])
        if total + tokens > max_tokens:
            break
        kept.insert(0, msg)
        total += tokens
    return kept

Summarization: compress older messages into a summary block.

def compress_history(messages: list, max_tokens: int) -> list:
    if count_tokens(messages) <= max_tokens:
        return messages
    old = messages[:-4]  # keep last 4 messages intact
    recent = messages[-4:]
    summary = llm.generate(f"Summarize this conversation:\n{format_messages(old)}")
    return [{"role": "system", "content": f"Previous conversation summary: {summary}"}] + recent

Token-aware truncation: drop middle messages, keep first (system context) and last (recency).

RAG context budgeting

Retrieved chunks consume budget fast. Control retrieval quality and quantity:

def retrieve_with_budget(query: str, budget_tokens: int, top_k: int = 10) -> str:
    chunks = retriever.search(query, top_k=top_k)
    selected, used = [], 0
    for chunk in chunks:
        tokens = count_tokens(chunk.text)
        if used + tokens > budget_tokens:
            break
        selected.append(chunk)
        used += tokens
    return format_chunks(selected)

Better retrieval beats more retrieval. Five highly relevant 200-token chunks outperform twenty mediocre 500-token chunks.

Output token limits

Set max_tokens per task type:

OUTPUT_LIMITS = {
    "classification": 50,
    "extraction": 500,
    "summarization": 1000,
    "chat": 2000,
    "code_generation": 4000,
}

For streaming responses, implement early stopping when the model produces a completion signal:

async def stream_with_limit(response, max_tokens: int):
    count = 0
    async for chunk in response:
        count += count_tokens(chunk.text)
        yield chunk
        if count >= max_tokens:
            break

Per-user and per-request rate limiting

Token budgets at the user level prevent abuse and runaway costs:

from datetime import datetime, timedelta

class UserTokenLimiter:
    def __init__(self, daily_limit: int):
        self.daily_limit = daily_limit
        self.usage = {}  # user_id → {date: tokens_used}

    def check_and_consume(self, user_id: str, tokens: int) -> bool:
        today = datetime.now().date()
        user_usage = self.usage.get(user_id, {})
        used_today = user_usage.get(today, 0)

        if used_today + tokens > self.daily_limit:
            return False
        user_usage[today] = used_today + tokens
        self.usage[user_id] = user_usage
        return True

Combine with request-level limits: max input tokens, max output tokens, max requests per minute.

Monitoring token consumption

Track these metrics in production:

• Input tokens per request (p50, p95, p99)
• Output tokens per request (p50, p95, p99)
• Cost per request (input + output × price per token)
• Budget utilization (% of context window used)
• Truncation rate (% of requests where input was truncated)

Alert when p95 input tokens exceed 80% of your budget — that means you are one bad RAG retrieval away from context overflow.

Common production mistakes

Teams get token budgeting strategies wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

LLM features around token budgeting strategies break in production when prompts assume deterministic output, context windows are sized for dev datasets, or token costs are never budgeted per user session. Always log prompt hash, model version, and latency—not raw prompts with PII.

Debugging and triage workflow

When token budgeting strategies misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• tiktoken tokenizer library (OpenAI)
• Anthropic token counting guide
• LangChain conversation buffer window memory
• OpenAI rate limits and usage tiers
• LLM cost calculator and comparison tools