Multimodal Embeddings with CLIP

By Michael Samuel Naeem · December 10, 2025 · 5 min read

A user types "matte black wireless earbuds case open" and expects your catalog search to surface the right SKU — from product photos alone, no tags required. CLIP-style models make that possible by embedding images and text into the same vector space, trained on hundreds of millions of image-caption pairs so "semantic similarity" aligns with human association more than color histograms ever could. The architecture is simple; the engineering is indexing millions of images, picking the right checkpoint, and knowing where zero-shot CLIP stops and fine-tuning begins.

Architecture: dual encoders, contrastive loss

Image ──► Image Encoder ──► L2 normalize ──► v_img ──┐
                                                      ├── cosine similarity
Text  ──► Text Encoder  ──► L2 normalize ──► v_txt ──┘

Training batch of N pairs treats N²−N implicit negatives. At inference, encode once, search with approximate nearest neighbors.

import open_clip
import torch
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-B-32", pretrained="laion2b_s34b_b79k"
)
tokenizer = open_clip.get_tokenizer("ViT-B-32")

image = preprocess(Image.open("product.jpg")).unsqueeze(0)
text = tokenizer(["wireless earbuds in open charging case"])

with torch.no_grad():
    img_feat = model.encode_image(image)
    txt_feat = model.encode_text(text)
    img_feat /= img_feat.norm(dim=-1, keepdim=True)
    txt_feat /= txt_feat.norm(dim=-1, keepdim=True)
    sim = (img_feat @ txt_feat.T).item()

Higher cosine similarity indicates alignment. Thresholds are dataset-specific — calibrate on validation queries.

Zero-shot classification

Encode class prompts as text prototypes:

labels = ["running shoes", "sandals", "boots", "loafers"]
text_tokens = tokenizer([f"a photo of {l}" for l in labels])
text_features = model.encode_text(text_tokens)
text_features /= text_features.norm(dim=-1, keepdim=True)

logits = (100.0 * img_feat @ text_features.T).softmax(dim=-1)

No classifier head retraining — useful for rapid taxonomy experiments. Production catalogs with fine-grained SKU distinctions usually outgrow zero-shot accuracy.

Building image search index

Offline pipeline:

1. Resize/crop consistently (center crop vs pad affects fashion)
2. Batch encode_image on GPU
3. Store vectors + product IDs in FAISS/HNSW
4. Optional: store text embeddings from generated captions for hybrid retrieval

Query path: encode text → ANN search → rerank with metadata filters (price, stock).

For reverse image search, encode query image and search the image index directly — useful for duplicate detection and "shop the look."

Caption generation vs CLIP alone

Some pipelines generate alt-text with a VLM, then embed text with a text-only model. CLIP skips the caption bottleneck but misses OCR text in images (screenshots, labels). Hybrid approach:

• CLIP image vector
• OCR text embedded separately
• Concatenate or weighted fusion at rerank stage

Test which failure mode hurts your catalog more: visual similarity without words, or words without visual nuance.

Fine-tuning cautions

Fine-tune with image-text pairs from your domain using contrastive loss in OpenCLIP training scripts. Use:

• Low learning rate (1e-6 to 1e-5 for encoders)
• Mix 10–30% general LAION-style pairs to preserve alignment
• Early stopping on retrieval recall, not training loss

Avoid overfitting to studio backgrounds — augment with crop, color jitter, and random JPEG compression matching user-upload quality.

Production indexing pipeline

End-to-end CLIP search deployment:

Catalog images → batch encode (GPU cluster) → vector DB (FAISS/Pinecone/pgvector)
User query text → encode text → ANN search → metadata filter → rerank → results

Batch encoding throughput matters at scale — ViT-B-32 processes ~500 images/sec on A100. Plan GPU capacity for initial index build and nightly incremental updates for new products.

# Batch indexing pattern
def index_catalog(image_paths: list[str], batch_size: int = 64):
    vectors = []
    for batch in chunked(image_paths, batch_size):
        images = torch.stack([preprocess(Image.open(p)) for p in batch])
        with torch.no_grad():
            feats = model.encode_image(images)
            feats /= feats.norm(dim=-1, keepdim=True)
        vectors.extend(feats.cpu().numpy())
    index.add(np.array(vectors))

Store product metadata separately — ANN returns IDs, application layer joins price/stock/availability.

Evaluation before production

Build a retrieval eval set before launching:

# Golden set: query_text → relevant_product_ids
eval_set = [
    {"query": "black running shoes", "relevant": ["sku-123", "sku-456"]},
    {"query": "wireless charger stand", "relevant": ["sku-789"]},
]

def recall_at_k(queries, index, k=10):
    hits = 0
    total = 0
    for item in eval_set:
        results = search(item["query"], index, k=k)
        if any(r in item["relevant"] for r in results):
            hits += 1
        total += 1
    return hits / total

Target recall@10 > 0.85 on your eval set before shipping. Zero-shot CLIP often achieves 0.6–0.7 on domain-specific catalogs — fine-tuning or hybrid retrieval closes the gap.

Hybrid retrieval pattern

CLIP alone misses exact SKU matches and OCR text. Combine:

1. CLIP text→image — semantic similarity (primary)
2. BM25 on product title/description — keyword matching
3. Exact SKU filter — when query matches SKU pattern

Rerank combined results with learned weights or simple score fusion. See Elasticsearch relevance tuning for BM25 side.

Failure modes

• Zero-shot on fine-grained catalog — "nike air max 90" vs "nike air max 95" confusion; need fine-tuning or metadata
• Studio vs user-upload photo gap — CLIP trained on web photos; user-uploaded product images differ
• No eval set — launch with unknown recall; discover bad search in production
• OCR text ignored — product labels in images invisible to CLIP; add OCR pipeline
• Catastrophic forgetting on fine-tune — model loses general alignment; mix generic pairs

Production checklist

• Eval set with recall@10 measured before launch
• Batch indexing pipeline with incremental update for new products
• Hybrid retrieval (CLIP + BM25 + SKU exact match)
• OCR pipeline for text-heavy product images
• Fine-tuning only after zero-shot eval shows gap
• Vector index sized for catalog growth with HNSW parameters tuned

Resources

• OpenCLIP GitHub repository
• Learning Transferable Visual Models (CLIP paper)
• SigLIP — sigmoid loss for language-image pre-training
• LAION datasets
• FAISS vector search