Temperature and Sampling, Demystified

By Michael Samuel Naeem · April 2, 2025 · 4 min read

You set temperature to 0.7 because the blog post said to. Your chatbot gives different answers to the same question. Your extraction pipeline occasionally returns malformed JSON. Your creative writing feature produces the same three opening sentences repeatedly.

Temperature and sampling parameters control how the model chooses the next token from its probability distribution. They are not magic knobs — they are precise mathematical operations on logits. Understanding what they do lets you pick the right settings per task instead of copying defaults.

From logits to tokens

At each generation step, the model outputs a logit vector — one score per token in the vocabulary (50,000–128,000 entries). Sampling converts logits to a token:

logits → softmax(temperature) → filter (top-k/top-p) → sample → next token

Every parameter in that pipeline changes which tokens are eligible and how likely each is.

Temperature: sharpening or flattening the distribution

Temperature divides logits before softmax:

import torch
import torch.nn.functional as F

logits = model_output  # shape: [vocab_size]

def sample_with_temperature(logits, temperature=1.0):
    scaled = logits / temperature
    probs = F.softmax(scaled, dim=-1)
    return torch.multinomial(probs, num_samples=1)

# temperature = 0.01 ≈ greedy (always picks highest)
# temperature = 1.0  = model's natural distribution
# temperature = 2.0  = flatter, more random

At temperature 0, "The capital of France is" almost always produces " Paris". At temperature 1.5, it might produce " Lyon", " a city", or " located".

Top-k: hard cap on candidates

Top-k keeps only the K highest-probability tokens and zeroes the rest:

def top_k_filter(logits, k=40):
    top_k_values, top_k_indices = torch.topk(logits, k)
    filtered = torch.full_like(logits, float('-inf'))
    filtered.scatter_(0, top_k_indices, top_k_values)
    return filtered

With k=40, the model samples only from the 40 most likely tokens. This prevents low-probability tail tokens (which cause hallucinations and incoherence) from being selected.

Typical values: k=40–50 for general use, k=1 equivalent to greedy.

Top-p (nucleus sampling): adaptive filtering

Top-p selects the smallest token set whose cumulative probability exceeds p:

def top_p_filter(logits, p=0.9):
    sorted_logits, sorted_indices = torch.sort(logits, descending=True)
    cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)

    # Remove tokens beyond the nucleus
    sorted_indices_to_remove = cumulative_probs > p
    sorted_indices_to_remove[1:] = sorted_indices_to_remove[:-1].clone()
    sorted_indices_to_remove[0] = False

    filtered = logits.clone()
    filtered[sorted_indices[sorted_indices_to_remove]] = float('-inf')
    return filtered

When the model is confident (one token at 95%), top-p=0.9 might keep only 2–3 tokens. When uncertain (flat distribution), it might keep 500+. This adaptiveness is why top-p generally outperforms fixed top-k.

Most production configs use top-p=0.9 or top-p=0.95 combined with temperature.

Frequency and presence penalties

These modify logits based on tokens already generated:

# Conceptual: reduce logit of tokens that already appeared
for token_id in generated_tokens:
    if token_id in logits:
        count = generated_tokens.count(token_id)
        logits[token_id] -= frequency_penalty * count  # per occurrence
        logits[token_id] -= presence_penalty              # flat penalty if seen

• Frequency penalty (0.0–2.0): reduces probability proportional to how many times a token already appeared. Higher values discourage repetition.
• Presence penalty (0.0–2.0): flat reduction if the token appeared at all. Encourages topic diversity.

Use frequency penalty 0.3–0.6 when the model repeats phrases in long outputs. Avoid high penalties on extraction tasks — they can suppress valid repeated field names.

Recommended settings by task

TASK_CONFIGS = {
    "extraction": {"temperature": 0.0, "top_p": 1.0},
    "classification": {"temperature": 0.0, "top_p": 1.0},
    "code_generation": {"temperature": 0.2, "top_p": 0.95},
    "summarization": {"temperature": 0.3, "top_p": 0.9},
    "chatbot": {"temperature": 0.7, "top_p": 0.9},
    "creative_writing": {"temperature": 0.9, "top_p": 0.95, "frequency_penalty": 0.5},
}

Do not tune these in isolation. Temperature and top-p interact — low temperature with low top-p is nearly greedy; high temperature with high top-p produces chaos.

Reproducibility in production

For deterministic pipelines:

# OpenAI
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[...],
    temperature=0,
    seed=42,  # fixed seed for reproducibility
)

# vLLM
params = SamplingParams(temperature=0.0, seed=42)

Log the full sampling configuration with every request. When output quality shifts after a model update, you need to know exactly which parameters were used to isolate whether the model or the settings changed.

Common production mistakes

Teams get temperature sampling explained wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

LLM features around temperature sampling explained break in production when prompts assume deterministic output, context windows are sized for dev datasets, or token costs are never budgeted per user session. Always log prompt hash, model version, and latency—not raw prompts with PII.

Resources

• The Curious Case of Neural Text Degeneration (top-p paper)
• OpenAI API sampling parameters
• Hugging Face generation strategies guide
• vLLM SamplingParams documentation
• Holtzman et al. on repetition in neural text generation