Document Parsing Pipelines for RAG

By Michael Samuel Naeem · February 6, 2026 · 6 min read

Retrieval quality in a RAG system is capped by the quality of the text you fed it, and that text comes out of a document parsing pipeline — the ingestion stage that converts PDFs, scans, HTML, and Office files into clean, chunk-ready text. It's the part nobody demos and the part that quietly decides whether your whole system works. I've watched teams pour weeks into rerankers and prompt tuning while their PDFs were being extracted as a soup of jumbled columns, and no amount of downstream cleverness rescues that.

The uncomfortable truth: for most enterprise RAG, parsing is 70% of the effort and 90% of the failures. Let's treat it with the seriousness it deserves.

Why "just extract the text" fails

A PDF doesn't store a document; it stores instructions for painting glyphs at coordinates. There is no inherent notion of paragraphs, reading order, or "this cell belongs to that row." Run a naive extractor on a two-column research paper and you get lines interleaved across columns, headers glued to body text, and footnotes injected mid-sentence. Feed that to a chunker and every chunk is contaminated.

The categories of pain, roughly in order of how often they bite:

• Reading order in multi-column and sidebar layouts.
• Tables, where structure is the meaning and flattening to text destroys it.
• Scanned pages, which are images and need OCR before anything else.
• Headers, footers, page numbers repeating into every chunk as noise.
• Figures and captions, where the caption is gold and the figure is unparseable.

Name these explicitly, because your pipeline needs a deliberate answer for each — not a single library call and a prayer.

A staged pipeline beats a single tool

No one parser handles everything, so I build parsing as stages, each with a fallback. The shape that has held up for me:

def parse_document(path: str) -> list[Element]:
    kind = classify(path)                    # digital PDF? scan? html? docx?
    if kind == "digital_pdf":
        elements = layout_parse(path)        # layout-aware extractor
        if looks_garbled(elements):          # heuristics: low alpha ratio, no spaces
            elements = vision_parse(path)     # escalate hard pages to a VLM
    elif kind == "scan":
        elements = ocr_parse(path)
    else:
        elements = native_parse(path)        # html/docx have real structure
    return normalize(elements)               # strip headers/footers, tag types

The classify step matters more than the parser you pick, because routing lets you spend money only where it's needed: cheap extraction for clean digital PDFs, OCR for scans, and an expensive vision-language model only for the pages that come out garbled. Escalation on demand keeps the average cost sane while still handling the ugly 5% that would otherwise poison retrieval.

Tables are their own problem

Tables are where flattening does the most damage. "Revenue 2024 120 2025 145" is useless; the model needs to know 120 is 2024's revenue. Two approaches work in practice, and I use both depending on the table:

1. Structured extraction to Markdown or HTML. Keep the grid. Markdown tables survive chunking well and the model reads them natively. This is the default for well-behaved tables.
2. Row-wise linearization. Turn each row into a self-contained sentence — "In 2024, revenue was 120M." — which embeds and retrieves better for lookup-style questions because each fact carries its own context.

For dense financial or scientific tables, a layout-aware or vision model that emits HTML preserves merged cells and spans that Markdown can't. Whatever you choose, keep the table as one retrievable unit with a caption or nearby heading attached, so a chunk about a table also carries what the table is about.

Chunk on structure, carry the metadata

Once extraction is clean, chunking gets far easier — and this is where parsing pays off. Because you tagged elements by type (heading, paragraph, table, list), you can chunk on semantic boundaries instead of blindly slicing every N characters. Split at heading boundaries, keep a table with its caption, never cut a code block in half.

Just as important, attach metadata to every chunk during parsing: source file, page number, section heading, document type, and a stable id. That metadata powers citations, filtering, and debugging later. The downstream craft of sizing chunks, overlapping them, and reranking is a whole discipline of its own — I cover it in RAG in production: chunking, reranking, and evals — but none of it helps if the text arriving at the chunker is already scrambled.

Retaining structural metadata also unlocks better retrieval strategies downstream. Section headings and titles are strong keyword signals, which is one reason parsing quality feeds directly into hybrid search combining BM25 and vectors: the sparse side leans on the exact terms your parser preserved.

OCR vs vision models: pick by page, not by policy

There's a tempting binary — "we use OCR" or "we use a vision model" — but the right answer is per-page routing.

OCR is fast and nearly free; a VLM is slow and pricey but understands structure a text extractor can't. The trap with vision models specifically is that they can hallucinate — invent a plausible number in a blurry cell — which OCR never does, it just fails visibly. So validate VLM output on tables against sanity checks (row/column counts, numeric totals) rather than trusting it blindly.

Measure parsing, not just retrieval

The mistake that costs the most time is having no visibility into parse quality. You tune retrieval, see poor results, and blame the retriever when the real problem is upstream. Instrument the parser: sample extracted pages and diff them against the source, track the fraction of pages that escalated to the expensive path, flag documents with suspiciously low text yield or broken tables.

Keep a small set of "known hard" documents — the multi-column paper, the financial statement, the crooked scan — and re-run them whenever you change a parser or bump a library version, so a "harmless" dependency upgrade doesn't silently regress extraction for a whole document class.

Parsing will never be the exciting part of your RAG stack, and that's exactly why it's neglected and why fixing it moves the needle more than another reranker. Clean, structured, well-tagged text is the foundation; build the pipeline that produces it deliberately, and the rest of RAG gets dramatically easier.

Resources

• Unstructured — open-source document ingestion
• PyMuPDF documentation
• Tesseract OCR engine
• Docling — document parsing toolkit
• LlamaIndex — data loading and parsing
• Camelot — table extraction from PDFs