ColPali — Treat PDF Pages as Images, Skip OCR Entirely¶
What problem does this solve?¶
PDF retrieval pipelines extract text and lose everything else. Tables collapse into linear reading order. Figures become caption-only references. Equations turn into garbled symbol streams or vanish completely. The expensive parts of the document — the layout, the diagrams, the structured tables — disappear before retrieval ever begins. ColPali (Faysse et al., 2024) bypasses extraction entirely. Each PDF page is rendered to an image, then embedded by a vision-language model with ColBERT-style late interaction. Retrieval works on patch-level vectors with MaxSim aggregation; queries match the visual structure of pages, not extracted text. State-of-the-art retrieval on PDF benchmarks, no OCR pipeline required, and the model sees figures and tables as first-class content.
Where it came from¶
ColPali was published in mid-2024 by Manuel Faysse and colleagues. The paper showed that page-as-image retrieval beat every text-extraction pipeline on the ViDoRe benchmark — by margins as large as 20-30 points on table-heavy and figure-heavy documents. The team open-sourced the model (vidore/colpali-v1.3) and colpali-engine, the Python library for using it.
By 2026 page-as-image retrieval has become the default for visual PDFs. Variants like ColQwen2.5 and ColNomic-3b push quality further; the underlying pattern is unchanged. The cookbook uses ColPali v1.3 because it's the smallest and easiest to run on a notebook-sized GPU.
Where it fits in the RAG landscape¶
Three approaches to PDF retrieval in 2026:
- Text-extraction RAG. OCR + text chunking + dense retrieval. Loses structure.
- ColPali / ColQwen / ColNomic (this recipe). Render pages, embed with VLM, late interaction. Preserves structure.
- Hybrid extraction + multimodal. Extract what extracts well; use multimodal for the rest. More complex.
Production pipelines mix: text extraction for clean prose, ColPali for tables and figures. The cookbook recipe shows the pure-multimodal path; the hybrid is a composition of recipe 35 plus the text recipes.
When to use it (and when not to)¶
Use ColPali for any corpus where visual structure carries information. Financial filings, scientific papers, slide decks, technical manuals — all good fits because their layout and figures are first-class content. Skip it for born-digital text (Markdown, HTML, plain text). The visual channel adds nothing and the latency cost is real. Skip it when you can't run a GPU. CPU inference works but is slow enough to be impractical at notebook scale. Skip it for very large corpora where storage is the bottleneck. Multi-vector indexes are ~30x bigger than single-vector dense.
The intuition¶
Four intuitions to carry:
Pages are the unit of retrieval. Not chunks, not tokens — entire rendered pages. The model encodes each page into a grid of patch vectors.
Late interaction is the same as ColBERT (Recipe 19). Query text gets per-token vectors; page gets per-patch vectors; MaxSim aggregates. The same algorithm, different modality.
No OCR is the headline feature. Tables, figures, equations, handwritten notes — everything visible on the page is in the embedding. Nothing is lost to extraction failures.
Storage explodes. Each page is roughly 1024 patch vectors. A 1000-page corpus is a million-vector index. Qdrant and Vespa handle this natively; production usually pairs with Matryoshka (Recipe 12) for cost.
Architecture¶
flowchart LR PDF[PDF document] --> R[Render each page
as PNG] R --> V[Vision LM
ColPali v1.3] V --> P[Patch vectors
per page] P --> S[(Multi-vector store)] Q[Query text] --> T[Query token
vectors] T --> MS[MaxSim aggregation] S --> MS MS --> A[Top-k pages]
References¶
- 📄 ColPali — Efficient Document Retrieval with Vision Language Models (Faysse et al., 2024) — The original paper.
- 💻 illuin-tech/colpali repository — Reference implementation and model weights.
- 📚 ViDoRe benchmark — The leaderboard where page-as-image methods dominate.
- 📄 ColBERT (Recipe 19) — The text-only ancestor of ColPali.
- 📚 Qdrant multi-vector documentation — The native storage path for ColPali in production.
- 💻 ColQwen2.5 / ColNomic-3b alternative checkpoints — Other page-as-image embedders.
Setup¶
Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')
from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing
client = LLMClient()
print(f'Provider: {client.provider} | Chat model: {client.chat_model}')
print(init_tracing())
Build the Pipeline, Step by Step¶
Step 1 — Render PDF pages to images¶
We use the cookbook's arXiv Mamba PDF. PyMuPDF renders each page to a PIL image at ~120 DPI; resolution above that wastes memory without quality gain on text-heavy pages.
# pip install 'rag-cookbook[multimodal]' before running this notebook.
import fitz
from PIL import Image
from io import BytesIO
from pathlib import Path
PDF = Path('../../corpus/arxiv-2403-mamba.pdf')
doc = fitz.open(PDF)
images = []
for i, page in enumerate(doc):
pix = page.get_pixmap(dpi=120)
img = Image.open(BytesIO(pix.tobytes('png'))).convert('RGB')
images.append((i + 1, img))
if len(images) >= 12:
break
print(f'Rendered {len(images)} pages')
Step 2 — Load ColPali¶
We load the v1.3 checkpoint from Hugging Face. ColPali takes a few seconds to load and ~4 GB of VRAM on a modest GPU; CPU works but is roughly 100x slower.
import torch
from colpali_engine.models import ColPali, ColPaliProcessor
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = ColPali.from_pretrained('vidore/colpali-v1.3', torch_dtype=torch.float32).to(device).eval()
processor = ColPaliProcessor.from_pretrained('vidore/colpali-v1.3')
print(f'ColPali loaded on {device}.')
Step 3 — Embed pages and a query¶
Pages get per-patch vectors; queries get per-token vectors. The processor's score_multi_vector runs the MaxSim aggregation.
with torch.no_grad():
page_inputs = processor.process_images([img for _, img in images]).to(device)
page_embs = model(**page_inputs)
print(f'Page embeddings: {page_embs.shape}')
q_inputs = processor.process_queries(['What is selective scan and why does it matter?']).to(device)
q_embs = model(**q_inputs)
print(f'Query embeddings: {q_embs.shape}')
Step 4 — Score with MaxSim¶
The score is a sum of per-query-token max similarities. Each query token finds its best matching page patch and contributes that match to the total.
with torch.no_grad():
scores = processor.score_multi_vector(q_embs, page_embs).cpu().tolist()[0]
ranked = sorted(zip(scores, [n for n, _ in images]), reverse=True)
for s, n in ranked[:5]:
print(f' page {n} score {s:.2f}')
Step 5 — Wrap as answer_question¶
The cookbook contract returns (answer, contexts). For ColPali, contexts are descriptions of the top-k pages — recipe 36 shows how to feed the images themselves to a VLM for the answer.
def answer_question(question: str) -> tuple[str, list[str]]:
with torch.no_grad():
qi = processor.process_queries([question]).to(device)
qe = model(**qi)
sc = processor.score_multi_vector(qe, page_embs).cpu().tolist()[0]
best = sorted(zip(sc, [n for n, _ in images]), reverse=True)[:3]
pages = [n for _, n in best]
return f'Top pages for visual retrieval: {pages}', [f'page {p}' for p in pages]
ans, _ = answer_question('How does Mamba's parallel scan work?')
print(ans)
Step 6 — Visualise the top retrieved page¶
Display the highest-scoring page image so you can see what ColPali matched on.
from IPython.display import display
best_idx = ranked[0][1] - 1 # back to 0-based
display(images[best_idx][1].resize((400, 600)))
Look Inside¶
Inspect — score distribution across all pages¶
How sharp is the ranking? A steep curve means ColPali confidently identified the right pages.
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6, 2.5))
ax.bar(range(1, len(scores) + 1), scores)
ax.set_xlabel('page')
ax.set_ylabel('MaxSim score')
ax.set_title('ColPali score distribution')
plt.tight_layout()
plt.show()
Inspect — multiple queries, multiple winners¶
Different queries should pick different pages. If everything routes to page 1, the model isn't differentiating.
for q in [
'What is HiPPO initialization?',
'Show the experimental results.',
'What is the abstract about?',
'What does the parallel scan diagram look like?',
]:
with torch.no_grad():
qi = processor.process_queries([q]).to(device)
qe = model(**qi)
sc = processor.score_multi_vector(qe, page_embs).cpu().tolist()[0]
best_page = sorted(zip(sc, [n for n, _ in images]), reverse=True)[0][1]
print(f' {q[:55]:55s} -> page {best_page}')
Inspect — storage cost¶
Multi-vector storage is ~30x single-vector. Quantify on the current page set.
single_vec_bytes = 1024 * 4 # typical text embedding
page_size = page_embs.shape[1] * page_embs.shape[2] * 4
print(f'Single-vector per page: {single_vec_bytes:,} bytes')
print(f'ColPali multi-vector page: {page_size:,} bytes')
print(f'Ratio: ~{page_size // single_vec_bytes}x')
Inspect — latency¶
Per-page embedding and per-query scoring; measure both.
import time
with torch.no_grad():
t0 = time.perf_counter()
_ = model(**processor.process_queries(['probe']).to(device))
q_ms = (time.perf_counter() - t0) * 1000
t0 = time.perf_counter()
_ = processor.score_multi_vector(q_embs, page_embs)
score_ms = (time.perf_counter() - t0) * 1000
print(f'Query embed: {q_ms:.1f} ms')
print(f'MaxSim score: {score_ms:.1f} ms')
Run It¶
End-to-end ColPali retrieval.
ans, _ = answer_question('Where does the paper show experimental complexity results?')
print(ans)
Side by Side: Vanilla Baseline vs This Technique¶
Text-only baseline vs ColPali. The interesting cases are pages with tables or figures that text extraction would mangle.
from cookbook.baselines import vanilla_pipeline
q = 'Where does the paper show experimental complexity results?'
base = vanilla_pipeline(q, corpus='arxiv-mamba', top_k=3)
ours_a, _ = answer_question(q)
import pandas as pd
pd.DataFrame([
{'pipeline': 'text-extraction', 'preview': base.answer[:140]},
{'pipeline': 'colpali', 'preview': ours_a[:140]},
])
Knobs to Turn¶
Six knobs in priority order:
- Model checkpoint. ColPali v1.3, ColQwen2.5, ColNomic-3b are all strong. Benchmark on your corpus.
- Render DPI. 120 is the cookbook default. Higher costs memory and storage; lower loses detail in dense text.
- Page count cap. Notebook demos cap at 12; production indexes thousands of pages per document. Plan for storage.
- GPU vs CPU. CPU works for tiny corpora; GPU is essentially required at scale. A modest 16 GB GPU handles ColPali v1.3 comfortably.
- Native multi-vector store. Qdrant, Vespa, Weaviate all have native multi-vector paths. Use them in production.
- Quantisation. Combine with Matryoshka (Recipe 12) or product quantisation to manage the multi-vector storage cost at scale.
Evaluate on a Slice¶
Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample
qs = load_eval_questions()
samples = []
for row in qs[:5]:
answer, contexts = answer_question(row['question'])
samples.append({
'question': row['question'],
'expected': row['answer'],
'actual': answer[:200],
'contexts_retrieved': len(list(contexts)),
})
import pandas as pd
pd.DataFrame(samples)
Closing Thoughts¶
Four failure modes:
- Memory limits. ColPali on a small GPU runs out of VRAM with 50+ pages in a batch. Stream and process in chunks.
- Slow CPU inference. A 25-page document takes 5+ minutes on CPU. Use GPU or accept the wait.
- No native answer generation. ColPali retrieves; it does not synthesise. Pair with recipe 36 (VLM synthesis) for end-to-end visual QA.
- Storage at scale. Multi-vector indexes are large. Combine with Matryoshka (Recipe 12) or product quantisation to manage cost.
Compose with recipe 36 (VLM synthesis over pages) for full end-to-end visual RAG, with ColBERT (Recipe 19) for the text-mode cousin, and with hybrid retrieval (Recipe 18) for corpora that mix visual and prose documents.