RAPTOR — Recursive Abstractive Trees¶
What problem does this solve?¶
Chunks are flat. A 384-token chunk represents 384 tokens. A question that asks "across the whole document, what is the recurring theme?" cannot be answered from any single chunk because no chunk is the recurring theme. The recurring theme lives one level above the chunks, in the document's abstract structure. RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) builds a tree. Leaves are chunks. The next level up is LLM-written summaries of clusters of leaves. The next level is summaries of summaries. Retrieval can land at any level. Specific questions hit leaves; abstract questions hit higher-level summaries.
Where it came from¶
RAPTOR was published at ICLR 2024 by Sarthi et al. at Stanford. The paper showed that retrieval over a tree beat retrieval over leaves on multi-hop benchmarks (QASPER, QuALITY) by 6–10 points. The clustering algorithm — Gaussian mixture with BIC for k selection — is the most complex part of the paper; the abstractive summarisation step is straightforward. The cookbook's implementation uses k-means with a small k as a simpler stand-in. The abstractive summarisation step matches the paper's prompt structure. Recall lifts are smaller than the paper reports because our corpus is smaller, but the technique reads the same way.
Where it fits in the RAG landscape¶
RAPTOR is the multi-level extension of parent-child (Recipe 9). Parent-child has two levels: child and parent. RAPTOR has many: leaves, level-1 summaries, level-2 summaries, ad infinitum (in practice 2–3 levels). The tree is built bottom-up via clustering and abstractive summarisation.
Related techniques in the cookbook:
- Document summary routing (Recipe 11). Top level is the document summary; lower levels are sub-chunks within the chosen document. Simpler than RAPTOR but only handles document-level routing.
- GraphRAG (Recipe 31). Replaces the linear hierarchy with a knowledge graph; community-level summaries are the equivalent of higher tree levels. Better for cross-document reasoning at heavy computational cost.
- Parent-child (Recipe 9). Two-level RAPTOR essentially. Cheaper, lower ceiling.
When to use it (and when not to)¶
Use RAPTOR when your queries mix specific and abstract. Research-paper Q&A, book-length corpora, anything where some questions are "what is X" and others are "how does the work fit together". Skip it for narrow corpora where every question is at the same level of abstraction. FAQ Q&A bots have nothing to gain from a hierarchy. Skip it when LLM summarisation cost dominates your budget. Building the tree costs roughly one LLM call per cluster per level — manageable for thousands of chunks, expensive at billions.
The intuition¶
Three intuitions:
Each tree level is a different lens. Leaves are word-level fact retrieval; level 1 is paragraph-level synthesis; level 2 is section-level themes. The retriever picks the level by matching the query's abstraction.
Abstraction is built by summarisation. Each summary node is generated by prompting an LLM with its children's text. The summary inherits the children's information but rewords it at a higher level. The vector for the summary lives in a different part of embedding space than the vectors for the children.
Clustering needs to find topical groups. K-means is a fine default; the paper's BIC-based GMM is slightly better but slower. The cookbook uses k-means with n_clusters=max(2, len(leaves)//6).
Architecture¶
flowchart TB L0[Leaf chunks] --> KC[K-means cluster] KC --> LS[LLM:
summarise each cluster] LS --> L1[Level 1 summaries] L1 --> KC2[K-means cluster] KC2 --> LS2[LLM:
summarise] LS2 --> L2[Level 2 summaries] L0 --> S[(Vector store
all levels)] L1 --> S L2 --> S Q[Query] --> R[Search all levels] S --> R
References¶
- 📄 RAPTOR — Recursive Abstractive Processing for Tree-Organized Retrieval (Sarthi et al., 2024) — The ICLR 2024 paper.
- 📚 LlamaIndex RAPTOR pack — Reference implementation.
- 💻 Stanford NLP Group RAPTOR repository — The paper's code.
- 💻 LangChain RAPTOR cookbook — LangChain's implementation walk-through.
- 📚 Document Summary Index (Recipe 11) — Related two-level pattern.
- 💻 GraphRAG (Recipe 31) — Graph-structured alternative to RAPTOR's tree.
Setup¶
Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')
from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing
client = LLMClient()
print(f'Provider: {client.provider} | Chat model: {client.chat_model}')
print(init_tracing())
Provider: nebius | Chat model: meta-llama/Llama-3.3-70B-Instruct Tracing disabled.
Build the Pipeline, Step by Step¶
Step 1 — Load a slice of the corpus¶
We use 8 Rust chapters. RAPTOR is LLM-heavy, so a small corpus keeps the bill modest.
from cookbook.corpora import load_rust_book
docs = list(load_rust_book())[:8]
print(f'{len(docs)} chapters.')
8 chapters.
Step 2 — Build the leaf chunks¶
Standard chunking for the leaves. Fixed-window is fine here; semantic chunks would also work.
from cookbook.chunkers import fixed_window
leaves = fixed_window(docs, target_tokens=400, overlap_tokens=40)
print(f'{len(leaves)} leaf chunks.')
25 leaf chunks.
Step 3 — Embed the leaves¶
We need the embeddings to drive clustering.
import numpy as np
leaf_vecs = np.asarray(client.embed([l.text for l in leaves]), dtype=np.float32)
print(f'Leaf vectors: {leaf_vecs.shape}')
Leaf vectors: (25, 4096)
Step 4 — Cluster and summarise at level 1¶
K-means with k chosen to be roughly 1/6 of the leaves. Each cluster becomes one level-1 summary.
from sklearn.cluster import KMeans
def cluster_and_summarise(texts, vectors, n_clusters):
km = KMeans(n_clusters=n_clusters, n_init=4, random_state=42).fit(vectors)
summaries = []
for c in range(n_clusters):
members = [t for t, lbl in zip(texts, km.labels_) if lbl == c]
if not members:
continue
sample = '\n\n'.join(members[:5])[:4000]
s = client.chat(
'Summarize the recurring themes across these passages in 4 sentences. '
'Stay faithful to the source.\n\n' + sample
)
summaries.append(s)
summary_vecs = np.asarray(client.embed(summaries), dtype=np.float32)
return summaries, summary_vecs
L1_n = max(2, len(leaves) // 6)
L1_summaries, L1_vecs = cluster_and_summarise([l.text for l in leaves], leaf_vecs, n_clusters=L1_n)
print(f'L1 summaries: {len(L1_summaries)}')
print()
print('First L1 summary:')
print(L1_summaries[0][:400])
L1 summaries: 4 First L1 summary: The passages introduce the Rust programming language, covering topics such as installation, basic programming concepts, and unique features like ownership. The chapters build upon each other, starting with the basics of variables, types, and functions, and progressing to more advanced topics like structs, packages, and modules. A common theme throughout the passages is the emphasis on understandin
Step 5 — Level 2 summaries¶
Summarise the level-1 summaries. The tree gets a third floor.
L2_n = max(2, len(L1_summaries) // 3)
L2_summaries, L2_vecs = cluster_and_summarise(L1_summaries, L1_vecs, n_clusters=L2_n)
print(f'L2 summaries: {len(L2_summaries)}')
print()
print(f'Tree: {len(leaves)} leaves -> {len(L1_summaries)} L1 -> {len(L2_summaries)} L2')
L2 summaries: 2 Tree: 25 leaves -> 4 L1 -> 2 L2
Step 6 — Index all levels in one store¶
Mix leaves and summaries in one Qdrant collection. Retrieval picks the best at any level.
from cookbook.stores import QdrantBackend
all_texts = [l.text for l in leaves] + L1_summaries + L2_summaries
all_vecs = np.vstack([leaf_vecs, L1_vecs, L2_vecs]).tolist()
all_ids = (
[f'L0-{i}' for i in range(len(leaves))] +
[f'L1-{i}' for i in range(len(L1_summaries))] +
[f'L2-{i}' for i in range(len(L2_summaries))]
)
store = QdrantBackend('raptor', dim=len(all_vecs[0]))
store.add(all_texts, all_vecs, ids=all_ids)
print(f'Indexed {len(all_texts)} nodes total.')
C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
Indexed 31 nodes total.
Step 7 — Search the tree¶
Standard search; the retrieval engine doesn't know the tree exists. It just picks the top-k vectors. Specific queries hit leaves; abstract queries hit summaries.
q = 'Across the chapters here, what is the general pattern Rust uses for optional or fallible computation?'
qv = client.embed([q])[0]
for h in store.search(qv, top_k=6):
print(f' {h.doc_id:8s} score={h.score:.3f} {h.text[:140]}')
L0-21 score=0.631 # Enums and Pattern Matching In this chapter, we’ll look at enumerations, also referred to as _enums_. Enums allow you to define a type by e L0-5 score=0.609 6][enums]<!-- ignore --> will cover enums in more detail. The purpose of these `Result` types is to encode error-handling information. `Resu L0-4 score=0.594 handle to get input from the user. We’re also passing `&mut guess` as the argument to `read_line` to tell it what string to store the user i L0-16 score=0.581 user inputs a non-number, let’s make the game ignore a non-number so that the user can continue guessing. We can do that by altering the lin L0-12 score=0.530 and the code that should be run if the value given to `match` fits that arm’s pattern. Rust takes the value given to `match` and looks throu L0-18 score=0.525 # Common Programming Concepts This chapter covers concepts that appear in almost every programming language and how they work in Rust. Many
Step 8 — Wrap as answer_question¶
Same cookbook contract.
def answer_question(question: str, k: int = 6) -> tuple[str, list[str]]:
qv = client.embed([question])[0]
hits = store.search(qv, top_k=k)
contexts = [h.text for h in hits]
answer = client.chat(
'Use these passages.\n' + '\n\n'.join(contexts) + f'\nQ: {question}\nA:'
)
return answer, contexts
ans, _ = answer_question('How does Rust express optional and fallible computation across the language?')
print(ans)
Rust expresses optional and fallible computation across the language through the use of enums, specifically the `Option` and `Result` enums.
- The `Option` enum is used to express a value that may or may not be present. It has two variants: `Some(value)` and `None`.
- The `Result` enum is used to express a value that may be either a success (`Ok(value)`) or a failure (`Err(error)`).
These enums are used extensively throughout the Rust standard library and are the foundation of Rust's error handling system. They allow developers to explicitly handle errors and optional values in a concise and expressive way, making Rust code more robust and reliable.
In addition to enums, Rust also provides other mechanisms for handling errors and optional values, such as pattern matching, `if let` statements, and the `?` operator. These mechanisms make it easy to write concise and expressive code that handles errors and optional values in a robust and reliable way.
For example, the `Result` enum is used to handle potential failures when reading input from the user, as shown in the following code:
```rust
use std::io;
let mut guess = String::new();
io::stdin().read_line(&mut guess)
.expect("Failed to read line");
```
In this example, the `read_line` function returns a `Result` value, which is then handled using the `expect` method. If the `read_line` function fails, the program will crash and display the error message "Failed to read line".
Similarly, the `Option` enum is used to handle optional values, such as when parsing a string to an integer:
```rust
let guess: u32 = match guess.trim().parse() {
Ok(num) => num,
Err(_) => continue,
};
```
In this example, the `parse` function returns an `Option` value, which is then handled using a `match` statement. If the `parse` function fails, the program will continue to the next iteration of the loop.
Overall, Rust's use of enums to express optional and fallible computation makes it a robust and reliable language that is well-suited for systems programming and other applications where error handling is critical.
Look Inside¶
Inspect — which level wins for which question?¶
Specific questions should hit leaves; abstract questions should hit summaries. Verify.
for q in [
'What is the difference between Rc and Arc?', # specific -> leaf
'What recurring patterns appear across the Rust chapters?', # abstract -> L1/L2
'How does Rust handle errors in general?', # mid -> L1
]:
qv = client.embed([q])[0]
top = store.search(qv, top_k=3)
print(f' {q[:60]}:')
for h in top:
print(f' {h.doc_id} score={h.score:.3f}')
What is the difference between Rc and Arc?:
L0-17 score=0.420
L0-11 score=0.418
L0-19 score=0.411
What recurring patterns appear across the Rust chapters?:
L0-18 score=0.551
L0-17 score=0.544
L0-19 score=0.528
How does Rust handle errors in general?:
L0-16 score=0.704
L0-5 score=0.695
L0-4 score=0.658
Inspect — distribution of hits by level¶
Across a battery of queries, count how often each level wins. A healthy tree has each level represented.
from collections import Counter
battery = [
'What is Rc?', 'When should I use Arc?', 'What is borrow checking?',
'How does Rust handle errors?', 'What patterns appear across the book?',
'How does Rust express concurrency?', 'What is a trait?',
'How is async different from threads?',
]
level_hits = Counter()
for q in battery:
qv = client.embed([q])[0]
top = store.search(qv, top_k=1)[0]
level_hits[top.doc_id.split('-')[0]] += 1
print(dict(level_hits))
{'L0': 8}
Inspect — read one level-1 summary¶
Sanity-check the summary quality. A bad summary will be retrieved but won't help generation.
import random
print(random.choice(L1_summaries))
The passages introduce the reader to the Rust programming language through a hands-on project, a guessing game, which demonstrates common Rust concepts such as variables, methods, and external crates. The project starts with setting up a new Rust project using Cargo, the Rust package manager, and generating a "Hello, world!" program. The reader is then guided through the process of compiling and running the program, and eventually, writing code to process user input and print output. Throughout the passages, the reader is encouraged to practice the fundamentals of Rust programming, with the guessing game serving as a classic beginner programming problem to illustrate key concepts and ideas.
Inspect — build cost¶
Count LLM calls and embeddings used to build the tree.
calls = len(L1_summaries) + len(L2_summaries)
embeds = len(leaves) + len(L1_summaries) + len(L2_summaries)
print(f'LLM summarisations: {calls}')
print(f'Total embeddings : {embeds}')
print('All cached, so re-running this notebook is free.')
LLM summarisations: 6 Total embeddings : 31 All cached, so re-running this notebook is free.
Run It¶
End-to-end on a synthesis question.
ans, ctxs = answer_question('What is the consistent way Rust expresses fallibility throughout the book?')
print('=== RAPTOR answer ===')
print(ans)
=== RAPTOR answer === The consistent way Rust expresses fallibility throughout the book is through the use of `Result` and `Option` enums, as well as error handling mechanisms such as `expect` and `match` expressions. This allows Rust to encode error-handling information and provide a way to handle potential failures in a explicit and safe manner.
Side by Side: Vanilla Baseline vs This Technique¶
Vanilla baseline vs RAPTOR. On a synthesis question, vanilla retrieves leaves and the model must synthesise; RAPTOR retrieves a higher-level summary that already did the synthesis.
from cookbook.baselines import vanilla_pipeline
q = 'What is the consistent way Rust expresses fallibility throughout the book?'
base = vanilla_pipeline(q, corpus='rust-book', top_k=5)
ours_a, ours_c = answer_question(q)
import pandas as pd
pd.DataFrame([
{'pipeline': 'vanilla', 'preview': base.answer[:160]},
{'pipeline': 'raptor', 'preview': ours_a[:160]},
])
| pipeline | preview | |
|---|---|---|
| 0 | vanilla | The passages do not contain the answer. |
| 1 | raptor | The consistent way Rust expresses fallibility ... |
Knobs to Turn¶
Four knobs:
- Number of levels. Two (leaves + L1) catches most of the value; three (L0/L1/L2) is the canonical RAPTOR shape. More than three is rarely useful because the top-level summaries become abstract enough to lose grounding.
- Cluster count per level. Heuristic:
len(parent_level) // 6for L1,// 3for L2. Tune by inspecting cluster coherence — if clusters are obviously mixing topics, lowern_clustersand re-run. - Summary prompt. Faithfulness is the priority. A summary that adds facts not in the source is poison for downstream generation. "Stay faithful to the source" in the prompt is not optional.
- Clustering algorithm. k-means is simple. The paper uses Gaussian mixture with BIC; the quality lift is small in practice.
Evaluate on a Slice¶
Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample
qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'rust-book']
samples = []
for row in qs[:5]:
answer, contexts = answer_question(row['question'])
samples.append({
'question': row['question'],
'expected': row['answer'],
'actual': answer[:200],
'contexts_retrieved': len(list(contexts)),
})
import pandas as pd
pd.DataFrame(samples)
| question | expected | actual | contexts_retrieved | |
|---|---|---|---|---|
| 0 | What is ownership in Rust? | A set of rules governing how memory is managed... | Ownership is Rust's most unique feature and ha... | 6 |
| 1 | What does the borrow checker do? | It statically enforces that references obey th... | The borrow checker is a key component of the R... | 6 |
| 2 | What is the difference between String and &str? | `String` is an owned, heap-allocated, growable... | The passages provided do not explicitly explai... | 6 |
| 3 | Describe how match is exhaustive in Rust. | The compiler requires `match` arms to cover ev... | In Rust, `match` is exhaustive, meaning that i... | 6 |
| 4 | What is a trait? | A trait is a named set of methods that types c... | A trait in Rust defines a set of methods that ... | 6 |
Closing Thoughts¶
Three failure modes:
- Summary hallucination. The summariser invents facts. Always check faithfulness on a sample before deploying.
- Cluster degeneracy. k-means with too-large k produces 1-leaf clusters. Cap k at
n_leaves / 4to avoid this. - Retrieval favours summaries. Summaries are short and well-formed, so they often score high on cosine. If your eval set is specific-question-heavy, this can dilute leaf retrieval. Tune by retrieving more total
kso leaves still get a seat.
Compose with semantic boundary chunking (Recipe 5) for the leaves and contextual headers (Recipe 7) on the leaves. RAPTOR is a multi-level retrieval architecture; the level-0 chunks themselves can be as fancy as your other recipes make them.