Temperature and Sampling at the Serving Layer

By Michael Samuel Naeem · June 1, 2025 · 7 min read

AILLMInferenceServing

Your A/B test showed temperature 0.3 beat 0.7 on offline evals. In production, p99 latency doubled and JSON extraction still flakes on Tuesdays.

How vLLM, Triton, and OpenAI-compatible APIs apply temperature, top-p, and seeds — and why identical prompts diverge across replicas. Notebook experiments rarely expose what production serving does: batched logits, fused kernels, per-request scheduling, and provider-specific interpretations of "temperature 0."

How serving stacks apply sampling

At inference, the model emits logits for the next token. The serving layer — vLLM, TensorRT-LLM, Triton with TensorRT backend, or a vendor API — applies temperature scaling before softmax, then optional top-k/top-p filtering, then sampling or greedy selection.

Greedy decoding (temperature → 0) is not identical across stacks. Some implementations use a small epsilon instead of true zero to avoid numerical edge cases. Batched requests with different temperatures may take different code paths, affecting determinism.

# Conceptual path — actual fusion happens in CUDA kernels
scaled = logits / max(temperature, 1e-5)
probs = softmax(scaled)
if top_p < 1.0:
    probs = nucleus_filter(probs, top_p)
token = argmax(probs) if temperature < 1e-3 else sample(probs)

Per-route policies beat global defaults

Production LLM products mix tasks on one cluster: JSON extraction, conversational chat, summarization, tool-call argument generation. Each needs different sampling:

Route	Temperature	Top-p	Rationale
Structured extraction	0.0	1.0	Minimize format variance
Customer chat	0.5–0.8	0.9–0.95	Natural phrasing, some variety
Brainstorm / marketing	0.9–1.1	0.95	Diversity over consistency
Tool args (JSON)	0.0–0.2	1.0	Valid schemas over creativity

Store policies in config, version them, and log prompt_version + sampling_policy_id with every completion. When extraction quality regresses, you need to know if the model or the policy changed.

Determinism, seeds, and "same prompt, different answer"

Customers expect temperature 0 to mean deterministic. In practice:

Batched inference reorders floating-point reductions across requests.
Different GPU/driver versions change numerics slightly.
Provider APIs may not honor seed on all models or may document "best effort" determinism.
Speculative decoding (draft model + verification) can change token selection unless disabled for eval runs.

For regression tests, pin model weights, container digest, CUDA version, and set seed where supported. Run golden tests on a dedicated single-request queue without batching if you need bit-stable outputs.

Latency interaction

Higher temperature does not directly increase latency, but sampling policies interact with stopping criteria and retry loops. Extraction at temperature 0.7 may produce invalid JSON → repair prompt → 2–3x tokens. That shows up as latency and cost, not as "temperature overhead."

Top-p/top-k add negligible compute relative to forward pass. The operational win is fewer retries by choosing the right policy upfront.

Operational checklist

Define sampling policies per route, not per environment only.
Log policy ID, model ID, seed (if any), and finish reason.
Alert when JSON-parse failure rate correlates with policy deploys.
Document provider-specific semantics for "temperature 0" in runbooks.
A/B test policies on live traffic with quality metrics, not only offline perplexity.

Tuning temperature in notebooks without per-route policies — creative chat and extraction share one global default. The fix is governance: treat sampling like API schema — reviewed, versioned, and rolled back independently of model weights.

Production hardening

Pin versions affecting serving-layer sampling parameters. Progressive rollout: internal tenants → canary → full promote. Keep previous config hot-swappable one release.

Handoff and ownership

Temperature and Sampling at the Serving Layer touches multiple teams — name DRIs in the service catalog. New hires should rollback safely using only the runbook within week one.

Operating serving-layer sampling parameters after scale events (review 1)

Traffic doublings, model swaps, and enterprise SSO enablement invalidate assumptions in the original design. Quarterly on-call reviews should update thresholds from recent incidents — not only the primary author's memory.

When temperature and sampling at the serving layer touches billing, auth, or retrieval, schedule a cross-team review after every major launch. Platform, product, security, and finance should agree on what the leading metric is and who owns rollback.

Game days to run: dependency slow-down, duplicate webhook delivery, index swap rollback, IdP cert rotation dry-run. Measure time-to-mitigate, not only time-to-detect. When providers change streaming or auth semantics without a deploy on your side, error-class metrics should catch drift within hours.