Batch Inference for Throughput

By Michael Samuel Naeem · October 22, 2024 · 5 min read

Running 50,000 product descriptions through an LLM for classification at $0.003 per call in real-time costs $150 and ties up rate limits your support bot needs during business hours. The same job via batch API costs $75, completes overnight, and never competes with production traffic. Batch inference is boring infrastructure — and it's how teams afford LLM features at scale.

Provider batch APIs

OpenAI and Anthropic both offer async batch endpoints: upload a JSONL file of requests, get results hours later at reduced price.

{"custom_id": "prod-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Classify: ..."}]}}
{"custom_id": "prod-002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Classify: ..."}]}}

async def submit_batch(requests: list[BatchRequest]) -> str:
    file_id = await client.files.create(
        file=to_jsonl(requests),
        purpose="batch",
    )
    batch = await client.batches.create(
        input_file_id=file_id,
        endpoint="/v1/chat/completions",
        completion_window="24h",
    )
    return batch.id

Poll or webhook on completion. Map results back via custom_id. Handle partial failures — batches can succeed 98% with individual line errors.

Best for: embeddings at ingest time, nightly report generation, eval dataset runs, reprocessing after prompt changes.

Self-hosted continuous batching

When provider batch isn't an option (private models, data residency, cost at volume), run vLLM or TGI with continuous batching:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.90

Continuous batching (also called in-flight batching) adds new requests to the GPU batch as others finish — unlike static batching where you wait for N requests before starting.

Throughput gains come from:

• Amortized attention computation across sequences
• Better GPU occupancy (fewer idle cycles)
• KV cache management via PagedAttention

Queue architecture for batch workloads

Separate batch from real-time at the queue level:

                    ┌──────────────┐
Producers ─────────→│  SQS/Kafka   │
                    └──────┬───────┘
                           │
              ┌────────────┼────────────┐
              ▼                         ▼
     [Real-time workers]        [Batch aggregator]
     latency SLO: 2s            flush every 30s OR 64 items
              │                         │
              ▼                         ▼
     [vLLM priority queue]      [vLLM batch queue]

Batch aggregator collects requests until size or time threshold, submits as a group. Real-time requests never wait behind batch jobs.

Sizing and backpressure

Monitor:

• Tokens/second per GPU — primary throughput metric
• Queue depth — growing depth means under-provisioned
• Batch fill rate — if batches are size 3 when target is 64, your aggregator window is too short or traffic is too sparse
• P99 latency for real-time — batch traffic shouldn't degrade this

Apply backpressure when queue depth exceeds threshold:

if queue.depth > MAX_DEPTH:
    raise ServiceUnavailable(retry_after=30)

For batch jobs, delay is acceptable. For real-time, shed load early.

Chunking large batch jobs

A 500K-row job shouldn't be one batch file:

• Split into chunks of 5,000–10,000 requests
• Submit chunks in parallel (respect provider concurrent batch limits)
• Checkpoint progress — store completed custom_ids so retries skip done rows
• Idempotent writes on the output side

for chunk in chunks(all_rows, size=5000):
    batch_id = await submit_batch(chunk)
    await db.save_checkpoint(job_id, batch_id, chunk.ids)

Cost comparison framework

Run the math before building real-time pipelines for inherently async work.

OpenAI Batch API workflow

Complete batch processing pipeline:

import openai, json, time

# 1. Prepare JSONL input file
requests = [
    {"custom_id": f"req-{i}", "method": "POST",
     "url": "/v1/chat/completions",
     "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": q}]}}
    for i, q in enumerate(queries)
]
with open("batch_input.jsonl", "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

# 2. Upload and create batch
file = openai.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch = openai.batches.create(input_file_id=file.id, endpoint="/v1/chat/completions", completion_window="24h")

# 3. Poll until complete
while batch.status not in ("completed", "failed"):
    time.sleep(60)
    batch = openai.batches.retrieve(batch.id)

# 4. Download results
results = openai.files.content(batch.output_file_id)

50% cost discount vs real-time. 24-hour completion window — not for latency-sensitive work.

Self-hosted batch inference with vLLM

For on-prem or cost control at scale:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B", max_num_seqs=256)
prompts = load_prompts("batch_job.csv")  # 100k prompts

sampling = SamplingParams(temperature=0, max_tokens=512)
outputs = llm.generate(prompts, sampling)  # continuous batching internally

save_results(outputs, "batch_output.parquet")

vLLM continuous batching processes requests as they complete — no fixed batch size. Throughput scales with GPU count linearly.

Batch vs real-time decision matrix

Default to batch for: nightly summaries, bulk classification, embedding generation, eval runs. Default to real-time for: chat, search, agent tools.

Failure modes

• Real-time for overnight work — 2× cost for no latency benefit
• Single giant batch file — failure loses all progress; chunk into 5k–10k
• No checkpoint on batch job — retry resubmits completed rows
• Batch for user-facing chat — 24h latency unacceptable
• No idempotent output writes — retry creates duplicate records

Production checklist

• Batch API used for all async workloads >10k requests/day
• Input chunked into 5,000–10,000 request files
• Checkpoint saved after each chunk submission
• Output writes idempotent (upsert on custom_id)
• Cost comparison documented: batch vs real-time per workload
• Batch job status monitored; alert on failed status

Resources

• OpenAI Batch API documentation
• Anthropic Message Batches API
• vLLM continuous batching paper
• Hugging Face Text Generation Inference
• NVIDIA TensorRT-LLM batching guide