Skip to content

Multimodal RAG

PDFs are not text. They are typeset documents with figures, tables, equations, and layout. Text-extraction pipelines silently lose all of that. Modern multimodal RAG embeds the pages, not the text, and lets a vision-language model read the picture.

The two recipes

  • ColPali Page-as-Image (35) — render PDF pages, embed with a ColBERT-style vision retriever (ColPali, ColQwen2.5, ColNomic), score with MaxSim. No OCR, no text extraction, full layout awareness.
  • VLM Synthesis Over Pages (36) — once ColPali has picked the right pages, the answer is generated by a VLM (Qwen2.5-VL, GPT-4o, Gemini 2.5) that sees the page rather than reading extracted text.

Why it wins

Three places text extraction loses:

  1. Tables. The reading order is wrong; relationships between cells are lost.
  2. Figures. Captions are extracted; the figure itself is not.
  3. Equations. Math layout collapses; LaTeX is rarely recovered.

ColPali + VLM retains all three.

The cost reality

Storage: ~1000 patch vectors per page. A 10,000-page corpus is 10M vectors. Manageable; use Qdrant's multi-vector path.

Compute: one VLM call per query at ~1500–4000 image tokens per page. Pick few pages with ColPali (top-3 is enough most of the time), then synthesize. Do not feed 50 pages to the VLM.

GPU: ColPali fits on a single 24 GB card for indexing; smaller for inference. CPU works but is slow.

When to use text instead

For docs that are born text (markdown, HTML, code) the visual channel adds nothing. Stick with text retrieval. ColPali is for PDFs that came from a typesetter, not from a Markdown export.

State of the art in 2026

The strongest open ColBERT-style vision retrievers are colpali-v1.3, colqwen2.5-v0.2, and colnomic-3b. The strongest open VLMs for synthesis are Qwen2.5-VL and Llama-3.2-Vision; hosted leaders are GPT-4o and Gemini 2.5. Benchmark on your own corpus — leaderboards disagree.