VLM Synthesis — Answering from Page Images¶
What problem does this solve?¶
ColPali retrieves pages but does not answer questions. You still need to generate an answer, and the answer should reflect what's visible on the page — including the tables and figures the visual retrieval surfaced. Text-only LLMs can't see images; the synthesis step has to use a vision-language model. Modern VLMs (Qwen2.5-VL, GPT-4o, Gemini 2.5, Llama-3.2-Vision) accept image inputs alongside text. The pattern: ColPali picks the relevant page images, send them plus the question to a VLM, get the answer. The VLM sees the tables, figures, and layout that text extraction would have lost.
Where it came from¶
The pattern emerged organically as VLMs became production-capable in 2024. GPT-4V (later GPT-4o) was the first frontier VLM available via API; Gemini Pro Vision followed. By mid-2025, open VLMs like Qwen2.5-VL and Llama-3.2-Vision had reached comparable quality. Combining them with ColPali-style retrieval (the upstream paper coined this composition) became the standard for visual document QA. By 2026 the composition is so common it has a name in production literature — "visual RAG" — and most multimodal RAG demos run this exact pattern. The cookbook factors retrieval (Recipe 35) and synthesis (this recipe) as separate notebooks so each step is individually inspectable, but in production they live as one pipeline.
Where it fits in the RAG landscape¶
Three approaches to visual document QA in 2026:
- Text-extraction + text LLM. Extracts text, answers from text. Loses visual content.
- VLM-only. Send the document straight to the VLM. Works for short documents; doesn't scale.
- ColPali + VLM (this recipe). Retrieve pages with ColPali, answer with VLM. Scales and preserves visual content.
The third pattern is the canonical production setup. The cookbook isolates the retrieval (Recipe 35) and synthesis (Recipe 36) steps so the parts are individually inspectable.
When to use it (and when not to)¶
Use VLM synthesis whenever your retrieved evidence is visual. ColPali outputs page images, financial documents, slide decks, scientific papers — all benefit from visual-aware answering. Skip it when your retrieval is text-only. There's nothing visual to synthesise from. Skip it when latency or cost matters more than visual fidelity. VLM calls are slower and more expensive than text-only calls because images cost a lot of input tokens.
The intuition¶
Four intuitions:
Image tokens are expensive. A 1024×1024 image is roughly 1500-4000 input tokens depending on the VLM. Five pages × 1500 tokens × $5/M input tokens = real money at scale.
The VLM sees layout. Tables, figures, equations, and spatial relationships all live in the image. The VLM uses them directly.
Pick few pages, not many. Sending 50 page images to the VLM is wasteful. ColPali picks the top-3 confidently; that's what the VLM should see.
Prompt for grounding. "Answer only from the visible content" reduces VLM hallucination noticeably. The model is otherwise tempted to reason beyond the image and import outside facts that may not be visible.
Architecture¶
flowchart LR CP[ColPali
top-3 pages] --> IMG[Page images] Q[Question] --> VLM IMG --> VLM[Vision LM] VLM --> A[Grounded answer]
References¶
- 📄 ColPali (Recipe 35) — The retrieval step upstream.
- 💻 Qwen2.5-VL model card — The open-weight VLM we recommend.
- 📚 OpenAI vision documentation — GPT-4o vision API.
- 📚 Anthropic vision documentation — Claude vision API.
- 📚 Gemini vision documentation — Gemini vision API.
- 📚 LiteLLM multimodal documentation — Cross-provider multimodal calls via LiteLLM.
Setup¶
Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')
from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing
client = LLMClient()
print(f'Provider: {client.provider} | Chat model: {client.chat_model}')
print(init_tracing())
Build the Pipeline, Step by Step¶
Step 1 — Reuse ColPali retrieval¶
This recipe builds on Recipe 35. The setup below reproduces ColPali retrieval; in production you'd reuse a shared state.
# pip install 'rag-cookbook[multimodal]' before running.
import fitz, torch
from PIL import Image
from io import BytesIO
from pathlib import Path
from colpali_engine.models import ColPali, ColPaliProcessor
PDF = Path('../../corpus/arxiv-2403-mamba.pdf')
doc = fitz.open(PDF)
images = []
for i, page in enumerate(doc):
pix = page.get_pixmap(dpi=120)
img = Image.open(BytesIO(pix.tobytes('png'))).convert('RGB')
images.append((i + 1, img))
if len(images) >= 12:
break
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = ColPali.from_pretrained('vidore/colpali-v1.3', torch_dtype=torch.float32).to(device).eval()
processor = ColPaliProcessor.from_pretrained('vidore/colpali-v1.3')
with torch.no_grad():
page_embs = model(**processor.process_images([img for _, img in images]).to(device))
print(f'ColPali ready: {len(images)} pages indexed.')
Step 2 — Pick top-k pages for a query¶
Same retrieval as recipe 35. We return PIL images for the top-k pages.
def colpali_pick(question: str, k: int = 3):
with torch.no_grad():
qe = model(**processor.process_queries([question]).to(device))
sc = processor.score_multi_vector(qe, page_embs).cpu().tolist()[0]
ranked = sorted(zip(sc, range(len(images))), reverse=True)[:k]
return [images[idx][1] for _, idx in ranked]
picks = colpali_pick('What is selective scan?')
print(f'Picked {len(picks)} pages.')
Step 3 — Encode images for the VLM¶
We base64-encode each PIL image. The LiteLLM-compatible message format accepts image_url with a data:image/png;base64,... URL.
import base64, io
def encode(img: Image.Image) -> str:
buf = io.BytesIO()
img.save(buf, format='PNG')
return base64.b64encode(buf.getvalue()).decode()
encoded = [encode(img) for img in picks[:2]]
print(f'Encoded {len(encoded)} pages, total size: ~{sum(len(e) for e in encoded) // 1024} KB')
Step 4 — Send to the VLM¶
Build a multimodal message: one text part with the question, one image part per picked page. The cookbook's client.chat supports a list-of-dicts payload directly.
def answer_visually(question: str, pages: list[Image.Image]) -> str:
parts = [{'type': 'text', 'text': f'Answer using only what is visible in these pages.\nQ: {question}'}]
for img in pages:
parts.append({
'type': 'image_url',
'image_url': {'url': f'data:image/png;base64,{encode(img)}'},
})
return client.chat([{'role': 'user', 'content': parts}])
ans = answer_visually('What is selective scan and why does it matter?', picks[:2])
print(ans[:400])
Step 5 — Wrap as answer_question¶
Cookbook contract. Internally we ColPali-pick then VLM-synthesise.
def answer_question(question: str) -> tuple[str, list[str]]:
pages = colpali_pick(question, k=3)
return answer_visually(question, pages), [f'page-image-{i}' for i in range(len(pages))]
ans, _ = answer_question('Explain the parallel scan implementation.')
print(ans[:400])
Look Inside¶
Inspect — what does the VLM see?¶
Display the top page image alongside the answer so you can verify the model is reading from the right source.
from IPython.display import display
display(picks[0].resize((400, 600)))
print('Top page above; VLM answered using this image as context.')
Inspect — image token cost¶
Each image is roughly 1500-4000 input tokens. Multiply by k pages for the per-query cost.
import sys
encoded_size = sum(sys.getsizeof(e) for e in encoded)
approx_tokens = encoded_size // 3 # base64 inflates ~33%; rough estimate
print(f'Encoded payload: {encoded_size:,} bytes')
print(f'Approximate input tokens: {approx_tokens:,}')
print('Tune k carefully to manage cost.')
Inspect — same question, different page counts¶
Compare answers from k=1, k=3, k=5 pages. More pages give the VLM more context but cost more.
q = 'What is selective scan?'
for k in (1, 3, 5):
pages = colpali_pick(q, k=k)
ans = answer_visually(q, pages)
print(f'k={k}: {ans[:200]}')
print()
Inspect — out-of-document question¶
Ask something the document doesn't cover. The VLM should refuse or hedge — "Answer only from the visible content" enforces this.
ans = answer_visually('Who painted the Sistine Chapel ceiling?', picks[:2])
print(ans[:300])
Run It¶
End-to-end visual QA.
ans, _ = answer_question('How does Mamba's complexity compare to attention?')
print('=== VLM-synthesised answer ===')
print(ans)
Side by Side: Vanilla Baseline vs This Technique¶
Text-extraction baseline vs ColPali + VLM. The contrast is clearest on questions whose answers live in figures or tables.
from cookbook.baselines import vanilla_pipeline
q = "How does Mamba's complexity compare to attention?"
base = vanilla_pipeline(q, corpus='arxiv-mamba', top_k=3)
ours_a, _ = answer_question(q)
import pandas as pd
pd.DataFrame([
{'pipeline': 'text-extraction', 'preview': base.answer[:160]},
{'pipeline': 'colpali + vlm', 'preview': ours_a[:160]},
])
Knobs to Turn¶
Six knobs in priority order:
- VLM choice. Qwen2.5-VL, GPT-4o, Gemini 2.5, Claude 3.5 Sonnet. Benchmark on your domain; image-handling varies a lot.
- Page count
k. Default 3. Higher catches more context, costs more tokens. Most queries are well-served by 2-3 pages. - Image resolution. 120 DPI is the cookbook default; lower if cost is tight, higher if fine details matter for the question.
- Prompt grounding. "Answer only from the visible content" reduces hallucination noticeably.
- Compose with text RAG. For born-digital documents, text RAG is cheaper; route per document type.
- Batch the VLM calls. When you have many similar queries against the same pages, batch them together to amortise the image-token cost.
Evaluate on a Slice¶
Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample
qs = load_eval_questions()
samples = []
for row in qs[:5]:
answer, contexts = answer_question(row['question'])
samples.append({
'question': row['question'],
'expected': row['answer'],
'actual': answer[:200],
'contexts_retrieved': len(list(contexts)),
})
import pandas as pd
pd.DataFrame(samples)
Closing Thoughts¶
Three failure modes:
- Cost explosion. Every page image is 1500-4000 input tokens. A k=10 query is roughly 30k input tokens, repeatedly. Plan budgets.
- VLM hallucination on weak images. Low-resolution pages or page images with little text confuse VLMs. Grade your inputs.
- Provider differences. Image-handling quality varies a lot between VLMs. Test on your domain before committing.
Compose with ColPali (Recipe 35) for retrieval. Compose with text retrieval recipes for hybrid mixed-modality corpora. Compose with hallucination guardrails (Recipe 40) for the production safety net.