Steering Output with Logit Bias

By Michael Samuel Naeem · December 18, 2024 · 5 min read

You need the model to respond with exactly "positive", "negative", or "neutral" — not "The sentiment is positive" or "Positive!". Prompting helps; logit bias guarantees the first token comes from your allowed set. Logit bias operates below the prompt layer, adjusting token probabilities at each generation step. It's a scalpel most teams don't know exists, useful for classification endpoints, format enforcement, and blocking specific outputs.

How logit bias works

During generation, the model computes logits (raw scores) for each token in the vocabulary. Logit bias adds a value before softmax sampling:

adjusted_logit[token_id] = original_logit[token_id] + bias_value

# Force response to start with valid JSON open brace
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Extract name and age as JSON"}],
    logit_bias={
        token_id("{"): 5,       # encourage
        token_id("```"): -100,  # ban markdown fences
    },
    max_tokens=100,
)

Bias -100 makes a token effectively impossible. Bias +5 to +20 nudges without forcing.

Getting token IDs

You need the tokenizer mapping:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
token_ids = {enc.encode(word)[0]: bias for word, bias in [
    ("positive", 10),
    ("negative", 10),
    ("neutral", 10),
    ("The", -100),
    ("I", -100),
]}

For multi-token words, bias each token ID individually. Single-token words in the vocabulary work most reliably.

Classification forcing

Restrict output to label vocabulary:

def classification_bias(labels: list[str], model: str) -> dict[int, float]:
    enc = tiktoken.encoding_for_model(model)
    bias = {}
    # Boost label tokens
    for label in labels:
        for token_id in enc.encode(label):
            bias[token_id] = 10
    # Suppress common preamble tokens
    for word in ["The", "Based", "I", "This", "It"]:
        for token_id in enc.encode(word):
            bias[token_id] = -50
    return bias

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": f"Classify: {text}"}],
    logit_bias=classification_bias(["billing", "technical", "other"], "gpt-4o-mini"),
    max_tokens=5,
    temperature=0,
)

Combine with temperature=0 for maximum determinism.

JSON key enforcement

Ensure specific keys appear:

JSON_KEYS = ['"name"', '"email"', '"phone"']
bias = {}
for key in JSON_KEYS:
    for tid in enc.encode(key):
        bias[tid] = 5
# Ban markdown
for tid in enc.encode("```"):
    bias[tid] = -100

Less reliable than constrained decoding for full schema compliance, but works on any model with logit_bias support.

Preventing unwanted patterns

Block specific outputs:

BLOCKED = ["As an AI", "I cannot", "I'm sorry"]
bias = {}
for phrase in BLOCKED:
    for tid in enc.encode(phrase):
        bias[tid] = -80

Heavy-handed — can cause unnatural phrasing. Prefer output filtering for safety; logit bias for format.

Limitations

• Multi-token sequences — biasing one token in a multi-token word doesn't guarantee the full word
• Context sensitivity — strong bias on common tokens ("I", "the") breaks grammar
• Not semantic — bias can't enforce "amount must be a number" — only token-level nudges
• Provider-specific — token IDs differ across models

For schema compliance, prefer structured outputs or constrained decoding. Use logit bias for simple, token-level steering.

Self-hosted logits processors

vLLM supports custom logits processors:

def allowed_token_processor(allowed_ids: set[int]):
    def processor(token_ids, logits):
        mask = torch.full_like(logits, float('-inf'))
        mask[allowed_ids] = 0
        return logits + mask
    return processor

Equivalent to logit bias with hard masking — useful for open-source deployment.

Token ID lookup workflow

Logit bias requires numeric token IDs, not strings:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
token_id = enc.encode("€")[0]  # get ID for euro symbol
print(f"Token ID for €: {token_id}")

# Build bias dict
logit_bias = {token_id: 100}  # strongly prefer €

Always verify token IDs match your deployment model — GPT-4 and GPT-4o may tokenize differently. Test bias effect on a sample prompt before production deployment.

Combining logit bias with temperature

High temperature randomizes token selection — logit bias becomes less effective:

# Weak steering: high temperature dilutes bias
response = client.chat.completions.create(
    model="gpt-4o",
    temperature=0.9,
    logit_bias={euro_token_id: 50},
    ...
)

# Strong steering: low temperature + high bias
response = client.chat.completions.create(
    model="gpt-4o",
    temperature=0.0,
    logit_bias={euro_token_id: 100},
    ...
)

For format enforcement (JSON braces, specific keywords), use temperature=0.0 with targeted bias. For soft nudges (prefer formal tone), temperature=0.3–0.5 with moderate bias (+20 to +50).

When to use logit bias vs alternatives

Logit bias is a scalpel for token-level nudges — not a replacement for structured outputs or prompt engineering.

Failure modes

• Bias on common tokens ("the", "I") — breaks grammar; test carefully
• Multi-token words — biasing first token doesn't guarantee full word
• Wrong token IDs — silent no-op if IDs don't match model tokenizer
• High temperature + strong bias — unpredictable interaction; use temperature=0 for format
• No fallback — bias fails silently; pair with output validation

Production checklist

• Token IDs verified against deployment model tokenizer
• Temperature=0 for format-critical bias applications
• Output validation after generation (don't rely on bias alone)
• Bias values tested on sample prompts before production
• Structured outputs preferred over logit bias for schema compliance
• Bias documented with reason (not magic numbers in code)

Common production mistakes

Teams get logit bias steering wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

LLM features around logit bias steering break in production when prompts assume deterministic output, context windows are sized for dev datasets, or token costs are never budgeted per user session. Always log prompt hash, model version, and latency—not raw prompts with PII.

Resources

• OpenAI logit_bias parameter docs
• tiktoken tokenizer library
• vLLM logits processors
• Hugging Face logits processor guide
• Outlines constrained generation (alternative)