Vision-Language Models in Production

By Michael Samuel Naeem · August 19, 2025 · 5 min read

A warehouse app needs to identify damaged packages from phone photos. A generic image classifier trained on ImageNet confuses "crushed corner" with "normal shadow." Vision-language models answer natural questions about images—"Is the shipping label torn? Is there liquid damage?"—without training a custom classifier per defect type. Getting VLMs into production means handling image tokens, latency, cost, and hallucination risk on visual claims.

Architecture patterns

Synchronous API: User uploads image → resize → VLM call → response. Works for <5 second SLA, low QPS.

Async queue: Upload to S3 → worker processes → webhook notifies. For batch catalog tagging, document review.

Edge + cloud hybrid: On-device model for coarse screening ("is this a package?"), cloud VLM for detailed analysis.

Image preprocessing pipeline

from PIL import Image
import io

def prepare_image(raw_bytes: bytes, max_dim: int = 1024) -> bytes:
    img = Image.open(io.BytesIO(raw_bytes))
    img = img.convert("RGB")
    if img.width > max_dim or img.height > max_dim:
        img.thumbnail((max_dim, max_dim), Image.LANCZOS)
    buf = io.BytesIO()
    img.save(buf, format="JPEG", quality=85, optimize=True)
    return buf.getvalue()

Log original and processed sizes. A 12 MB iPhone photo becoming 120 KB JPEG is normal and desirable.

Prompt patterns

Structured extraction:

Analyze this product image. Return JSON with:
- product_type (string)
- visible_damage (boolean)
- damage_description (string or null)
- label_readable (boolean)
Do not guess brand names you cannot clearly read.

Comparative reasoning:

Image 1 shows the item at receipt. Image 2 shows the item now.
List any new damage visible in Image 2 that was not present in Image 1.

Chain-of-thought (for complex scenes):

First describe what you see. Then answer: Does this scene show a fire hazard?
Base your final answer only on visible evidence.

Force JSON with response_format (OpenAI) or tool use (Claude) to simplify parsing.

Multi-image messages

GPT-4o and Claude accept multiple images per request—useful for before/after comparisons:

content = [
    {"type": "text", "text": "Compare these two package photos."},
    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64_before}"}},
    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64_after}"}},
]

Token cost scales with image count and resolution. Two 1024px images ≈ 2× the tokens of one.

Self-hosted deployment

vLLM with Qwen2-VL:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2-VL-7B-Instruct \
  --max-model-len 8192 \
  --limit-mm-per-prompt image=4

A single A100 handles 5–15 requests/second depending on image size. Quantize to AWQ or GPTQ for 2x throughput at minor quality loss.

Pin model revisions in your deployment manifest. Qwen2-VL-7B-Instruct v1 vs v2 produce different outputs on the same prompt.

Reducing hallucinations

VLMs confidently describe objects that aren't there. Mitigations:

• Ask for uncertainty: "If you cannot determine X, respond with 'unclear'."
• Require evidence: "Quote the visual feature that supports your answer."
• Ensemble: run two models; flag disagreements for human review.
• Confidence thresholds: if logprobs available, reject answers below threshold.

Latency optimization

Triage pattern: fast model classifies "damaged/not damaged" in 500 ms; detailed model runs only on positives.

Evaluation framework

@dataclass
class EvalCase:
    image_path: str
    question: str
    expected: str

def run_eval(model_fn, cases: list[EvalCase]) -> dict:
    correct = 0
    for case in cases:
        answer = model_fn(case.image_path, case.question)
        if normalize(answer) == normalize(case.expected):
            correct += 1
    return {"accuracy": correct / len(cases), "total": len(cases)}

Expand golden set monthly from production errors. VLMs improve with better prompts as much as better weights.

Multi-image and video inputs

Modern VLMs accept multiple images per request:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Compare damage before and after repair."},
            {"type": "image_url", "image_url": {"url": before_url}},
            {"type": "image_url", "image_url": {"url": after_url}},
        ],
    }],
)

Label images explicitly in prompt ("Image 1: before, Image 2: after") — models confuse order without guidance. Video frames: sample 1 fps for long clips, max 20 frames to control token cost.

Cost control

Image tokens dominate VLM bills:

Resize client-side before upload. Cache by (image_hash, prompt_hash) — product catalog queries hit the same images repeatedly.

Safety and content moderation

Run moderation API on user-uploaded images before VLM processing — prevents policy violations and reduces attack surface from adversarial images designed to extract system prompts.

Pair with multimodal document understanding for structured extraction from PDFs and scans.

Common production mistakes

Teams get multimodal vision language models wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Production implementations of multimodal vision language models fail when staging mirrors production topology poorly, rollback is untested, and on-call runbooks describe the happy path only.

Resources

• GPT-4o vision capabilities — image input limits and token counting
• Qwen2-VL GitHub — open-source VLM with dynamic resolution
• LLaVA project page — academic VLM benchmark leader
• vLLM multimodal docs — self-hosted VLM serving
• CLIP benchmark (LAION) — understanding vision encoder training data