Document Summary Routing — Find the Document First¶
What problem does this solve?¶
When your corpus is dozens of distinct documents (one document per topic), a query often only really cares about one document. Flat retrieval scatters chunks across all documents and pulls in distractors from neighbouring documents that happen to share vocabulary. The search space becomes dominated by topical noise, and the answer chunk has to win against thousands of other chunks instead of dozens. The pattern: build a per-document summary, index those summaries. At query time, retrieve the top-N documents from the summary index, then run chunk-level retrieval inside those documents only. The second retrieval is much sharper because the search space is 1–3 documents instead of dozens. The router pays one extra embedding lookup; in exchange the chunk retrieval works against an order of magnitude smaller pool.
Where it came from¶
LlamaIndex shipped DocumentSummaryIndex in mid-2023. The pattern existed in production systems before that — most enterprise search engines route on metadata or document classifiers — but LlamaIndex packaged the LLM-summary-as-router idea cleanly.
By 2025 it had become one of the standard moves for multi-document corpora. Combined with adaptive routing (Recipe 26), it lets a system handle queries across hundreds of distinct knowledge sources with one consistent stack.
Where it fits in the RAG landscape¶
Three routing strategies in this cookbook:
- Document summaries (this recipe). LLM-written one-paragraph summary per document. Retrieve by query. Best when documents are clearly topical.
- Semantic router (Recipe 17). Classifier with hand-curated example phrases per route. Cheaper, no LLM at query time, but the route definitions are hand-crafted.
- Metadata filter (Recipe 20). Structured filters on metadata fields. Free at query time, requires explicit metadata to exist on the chunks.
Stack them: semantic router to pick a coarse domain, document-summary to pick a specific document inside the domain, then chunk-level retrieval. Each layer cuts the search space by roughly an order of magnitude, which is why this combination scales so well to enterprise corpora.
When to use it (and when not to)¶
Use document summary routing when the corpus is dozens-to-thousands of separable documents. Wikipedia articles per topic, product knowledge bases per product, legal cases per case number. Skip it when your corpus is one long document (a book, a 10-K). There is only one document to route to. Skip it when document boundaries do not match query boundaries. If users routinely ask questions that span multiple documents, the routing layer becomes a bottleneck.
The intuition¶
Three intuitions:
The summary index is small. One vector per document. Even at 100k documents you have a tiny index that searches in microseconds. The cost is the one-time summarisation pass; everything afterwards is cheap.
The inner retrieval is sharp. Once you have narrowed to 1–3 documents, chunk-level retrieval is competing against tens of chunks, not tens of thousands. Recall jumps mechanically because the search space is smaller and the chunks are topically coherent.
Missing-the-router is unrecoverable. If the router routes wrong, no amount of inner retrieval helps. Always fall back to flat retrieval when the router score is below a threshold; the fallback is the safety net for queries that genuinely span documents or don't fit the corpus.
Architecture¶
flowchart TB Q[Query] --> R[Summary index
top-N documents] D[Documents] --> SUM[Per-doc summary] SUM --> R R --> CH[Chunks of the
chosen documents] CH --> S[Chunk retrieval] S --> A[Top-k chunks]
References¶
- 📚 LlamaIndex DocumentSummaryIndex — Reference implementation.
- 📚 LlamaIndex Router Query Engine — The general routing pattern.
- 📄 Adaptive RAG (Recipe 26) — Cousin technique — routes by query class instead of by document.
- 💻 Semantic Router library — The lighter-weight routing alternative used in Recipe 17.
- 📚 Hierarchical retrieval — LlamaIndex docs — General hierarchical retrieval patterns.
- 📄 RAPTOR (Recipe 10) — Different way to multi-level a corpus.
Setup¶
Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')
from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing
client = LLMClient()
print(f'Provider: {client.provider} | Chat model: {client.chat_model}')
print(init_tracing())
Provider: nebius | Chat model: meta-llama/Llama-3.3-70B-Instruct Tracing disabled.
Build the Pipeline, Step by Step¶
Step 1 — Load the corpus¶
Wikipedia superconductors is perfect: many articles, each on a distinct topic. A query about "Cooper pairs" really only cares about the Cooper-pair article and maybe BCS-theory.
from cookbook.corpora import load_wikipedia_superconductors
docs = list(load_wikipedia_superconductors())
print(f'{len(docs)} articles.')
42 articles.
Step 2 — Generate per-document summaries¶
One LLM call per document. We ask for a three-sentence summary; that is enough signal for routing without burning tokens. With caching on, re-running this is free.
summaries = []
for d in docs:
s = client.chat('Summarize this article in 3 sentences:\n\n' + d.text[:4000])
summaries.append((d, s.strip()))
print(f'Generated {len(summaries)} summaries.')
print()
print('First summary:')
print(summaries[0][1])
Generated 42 summaries. First summary: Here is a summary of the article in 3 sentences: The Bardeen-Cooper-Schrieffer (BCS) theory is a fundamental concept in physics that explains the phenomenon of superconductivity. According to the BCS theory, superconductivity occurs due to the condensation of pairs of electrons, known as Cooper pairs, which move through a lattice without resistance. This microscopic theory is able to explain many of the thermodynamic and electromagnetic properties of superconductors, providing a deeper understanding of this complex phenomenon.
Step 3 — Build the router (summary index)¶
Tiny store, one vector per document.
from cookbook.stores import QdrantBackend
sum_v = client.embed([s for _, s in summaries])
router = QdrantBackend('doc-summary', dim=len(sum_v[0]))
router.add(
[s for _, s in summaries],
sum_v,
ids=[d.doc_id for d, _ in summaries],
)
print(f'Router indexed.')
C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
Router indexed.
Step 4 — Build chunk retrieval inside selected documents¶
When the router picks documents, we chunk just those and build a small ad-hoc store. For very large corpora you would pre-build all per-document stores; for our notebook this on-the-fly approach is fine.
from cookbook.chunkers import sentence_window
by_id = {d.doc_id: d for d, _ in summaries}
def route_and_retrieve(question: str, top_docs: int = 2, top_chunks: int = 5):
qv = client.embed([question])[0]
doc_hits = router.search(qv, top_k=top_docs)
picks = [by_id[h.doc_id] for h in doc_hits if h.doc_id in by_id]
chunks = sentence_window(picks, sentences_per_chunk=4)
chunk_v = client.embed([c.text for c in chunks])
inner = QdrantBackend(f'routed-{abs(hash(question)) % 9999}', dim=len(chunk_v[0]))
inner.add([c.text for c in chunks], chunk_v, ids=[c.chunk_id for c in chunks])
return inner.search(qv, top_k=top_chunks), [d.doc_id for d in picks]
hits, picks = route_and_retrieve('How are SQUIDs used to detect tiny magnetic fields?')
print(f'Routed to: {picks}')
for h in hits:
print(f' {h.score:.3f} {h.text[:160]}')
Routed to: ['wiki:SQUID', 'wiki:Flux_qubit'] 0.755 # SQUID _Source: Wikipedia, CC BY-SA 4.0_ A SQUID is a very sensitive magnetometer used to measure extremely weak magnetic fields, based on superconducting lo 0.550 Orlando et al. at MIT in 1999 and fabricated shortly thereafter. During fabrication, the Josephson junction parameters are engineered so that a persistent curre 0.507 When the applied flux through the loop area is close to a half integer number of flux quanta, the two lowest energy eigenstates of the loop will be a quantum su 0.491 # Flux qubit _Source: Wikipedia, CC BY-SA 4.0_ In quantum computing, more specifically in superconducting quantum computing, flux qubits are micrometer sized 0.404 This separation, known as the "qubit non linearity" criteria, allows operations with the two lowest eigenstates only, effectively creating a two level system. U
Step 5 — Wrap as answer_question¶
Standard contract.
def answer_question(question: str) -> tuple[str, list[str]]:
hits, _ = route_and_retrieve(question)
contexts = [h.text for h in hits]
answer = client.chat(
'Use these passages.\n' + '\n\n'.join(contexts) + f'\nQ: {question}\nA:'
)
return answer, contexts
ans, _ = answer_question('What is the Meissner effect?')
print(ans)
The Meissner effect is the expulsion of a magnetic field from a superconductor during its transition to the superconducting state when it is cooled below the critical temperature, resulting in the repulsion of a nearby magnet.
Look Inside¶
Inspect — routing quality¶
For a battery of queries, see which documents got routed. Sanity-check by reading the document titles.
for q in [
'How are SQUIDs used for magnetometry?',
'What is BCS theory?',
'What is YBCO?',
'How does flux pinning work?',
]:
_, picks = route_and_retrieve(q)
print(f' {q} -> {picks}')
How are SQUIDs used for magnetometry? -> ['wiki:SQUID', 'wiki:Flux_qubit']
What is BCS theory? -> ['wiki:BCS_theory', 'wiki:Ginzburg–Landau_theory']
What is YBCO? -> ['wiki:YBCO', 'wiki:Cuprate_superconductor']
How does flux pinning work? -> ['wiki:Flux_pinning', 'wiki:Meissner_effect']
Inspect — what does a summary look like?¶
Read a few. Summary quality is the technique. If summaries are vague, routing fails.
import random
for d, s in random.sample(summaries, 3):
print(f' {d.doc_id}')
print(f' {s}')
print()
wiki:Iron-based_superconductor
Here is a summary of the article in 3 sentences:
Iron-based superconductors, discovered in 2006, are chemical compounds that contain iron and exhibit superconducting properties. The first iron-based superconductor, LaOFeP, had a low transition temperature, but subsequent compounds, such as fluorine-doped LaFeAsO, reached higher temperatures, including 26 K in 2008. Further research found that replacing certain elements in these compounds, such as lanthanum with other rare earth elements, could increase their superconducting temperature to as high as 52 K.
wiki:Cuprate_superconductor
Here is a summary of the article in 3 sentences:
Cuprate superconductors are a type of high-temperature superconducting material composed of layers of copper oxides and other metal oxides. The copper oxide layers are alternated with layers of other metal oxides, which serve as charge reservoirs. At normal pressure, cuprate superconductors have the highest known superconducting temperatures, making them a unique and significant class of materials.
wiki:Transmon
Here is a summary of the article in 3 sentences:
In quantum computing, a transmon is a type of superconducting charge qubit that is designed to be less sensitive to charge noise. The transmon was developed in 2007 by a team of researchers at Yale University and Université de Sherbrooke, led by Jens Koch and others. The name "transmon" is short for "transmission line shunted plasma oscillation qubit", which refers to its unique design featuring a Cooper-pair box with capacitively shunted superconductors to reduce noise sensitivity.
Inspect — fall-through threshold¶
If the router's top score is below a threshold, the right behaviour is to fall back to flat retrieval. Measure the threshold's effect on a query that genuinely doesn't fit any document.
qv = client.embed(['What is the boiling point of water?'])[0]
doc_top = router.search(qv, top_k=1)[0]
print(f'Top routing score for an out-of-corpus query: {doc_top.score:.3f}')
print('In production, treat anything below 0.4 as no-match.')
Top routing score for an out-of-corpus query: 0.378 In production, treat anything below 0.4 as no-match.
Inspect — recall@5 routed vs flat¶
Compare routed retrieval against flat retrieval across the eval set.
from cookbook.corpora import load_eval_questions
from cookbook.chunkers import fixed_window
qs = [q for q in load_eval_questions() if q['corpus'] == 'wikipedia-superconductors'][:8]
flat_chunks = fixed_window(docs, target_tokens=300, overlap_tokens=30)
flat_v = client.embed([c.text for c in flat_chunks])
flat_store = QdrantBackend('flat', dim=len(flat_v[0]))
flat_store.add([c.text for c in flat_chunks], flat_v, ids=[c.chunk_id for c in flat_chunks])
def recall(fn):
hits = 0
for q in qs:
out = fn(q['question'])
gold = [w.lower() for w in q['answer'].split() if len(w) >= 4]
if any(any(w[:6] in t.lower() for w in gold) for t in out):
hits += 1
return hits / max(1, len(qs))
def routed(q):
hits, _ = route_and_retrieve(q)
return [h.text for h in hits]
def flat(q):
qv = client.embed([q])[0]
return [h.text for h in flat_store.search(qv, top_k=5)]
print(f'flat recall@5 = {recall(flat):.2f}')
print(f'routed recall@5 = {recall(routed):.2f}')
flat recall@5 = 1.00
routed recall@5 = 0.88
Run It¶
End-to-end on a focused query.
ans, ctx = answer_question('What is BCS theory and which observations does it explain?')
print('=== Doc-summary routed answer ===')
print(ans)
=== Doc-summary routed answer === BCS theory, or Bardeen–Cooper–Schrieffer theory, is a microscopic theory of superconductivity. It explains many thermodynamic and electromagnetic properties of superconductors by describing superconductivity as a microscopic effect caused by a condensation of pairs of electrons known as Cooper pairs. These pairs move through the lattice without resistance.
Side by Side: Vanilla Baseline vs This Technique¶
Vanilla baseline (flat retrieval) vs document-summary routing.
from cookbook.baselines import vanilla_pipeline
q = 'What is BCS theory and which observations does it explain?'
base = vanilla_pipeline(q, corpus='wikipedia-superconductors', top_k=5)
ours_a, ours_c = answer_question(q)
import pandas as pd
pd.DataFrame([
{'pipeline': 'vanilla flat', 'preview': base.answer[:160]},
{'pipeline': 'routed', 'preview': ours_a[:160]},
])
| pipeline | preview | |
|---|---|---|
| 0 | vanilla flat | BCS theory, or Bardeen–Cooper–Schrieffer theor... |
| 1 | routed | BCS theory, or Bardeen–Cooper–Schrieffer theor... |
Knobs to Turn¶
Five knobs ranked by impact:
- Summary length. Three sentences is the cookbook default. Longer summaries route better — they mention more entities — at the cost of an inflated router index. Five sentences is a fine ceiling.
top_docsat the router. Default 2. More documents catch cross-document queries but pay for more chunking and embedding work at query time.- No-match threshold. The score below which the router falls back to flat retrieval. Default 0.4; tune on a labelled out-of-corpus slice. Too high means false negatives; too low means false positives.
- Summary prompt. Ask the model to mention the article's key entities by name. Vague summaries route badly.
- Summariser model. Cheaper than the answerer model is fine. The summaries are written once and reused for every query; spending budget once is fine.
Evaluate on a Slice¶
Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample
qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'wikipedia-superconductors']
samples = []
for row in qs[:5]:
answer, contexts = answer_question(row['question'])
samples.append({
'question': row['question'],
'expected': row['answer'],
'actual': answer[:200],
'contexts_retrieved': len(list(contexts)),
})
import pandas as pd
pd.DataFrame(samples)
| question | expected | actual | contexts_retrieved | |
|---|---|---|---|---|
| 0 | Who first observed superconductivity, and in w... | Heike Kamerlingh Onnes observed it in mercury ... | Heike Kamerlingh Onnes first observed supercon... | 4 |
| 1 | What is the Meissner effect? | The complete expulsion of magnetic flux from a... | The Meissner effect is the expulsion of a magn... | 5 |
| 2 | Distinguish Type-I from Type-II superconductors. | Type-I has a single critical field above which... | Type-I and Type-II superconductors can be dist... | 5 |
| 3 | What does BCS theory explain? | It explains conventional superconductivity thr... | BCS theory explains many thermodynamic and ele... | 2 |
| 4 | What is a Cooper pair? | Two electrons bound together by phonon exchang... | A Cooper pair is a pair of electrons bound tog... | 2 |
Closing Thoughts¶
Three failure modes:
- Bad summaries. A summary that fails to mention key entities loses them at routing. Re-summarise with a tighter prompt if you see specific entity-named queries miss the right document.
- Single-document corpora. No documents to route to. Pattern is wasted.
- Cross-document questions. "Compare YBCO and Meissner effect" wants both documents. Increase
top_docsto handle this; combined with sub-question decomposition (Recipe 16) for harder cases.
Compose with semantic routing (Recipe 17) for coarse-domain selection upstream and with reranking (Recipe 22) for the chunk-level retrieval downstream.