HyDE — Hypothetical Document Embeddings¶
What problem does this solve?¶
User queries are short. Documents are long. Their embeddings live in different parts of vector space, and cosine similarity often fails to bridge that gap. A query that asks "why does this model scale linearly?" rarely lands near a passage that answers that question — because answers are written in a different register, with more entities, with the topic stated as fact rather than as a question. HyDE (Hypothetical Document Embeddings) fixes the asymmetry by having the LLM write a fake answer first, then embedding that. The fake answer is in answer-register, so it lives close to real answers in embedding space. Retrieve against the hypothetical, return the actual passages it matches, generate the real answer. The first call is wasted compute; the retrieval lift typically justifies it.
Where it came from¶
HyDE was published by Luyu Gao and colleagues at CMU in late 2022 ("Precise Zero-Shot Dense Retrieval without Relevance Labels"). The paper showed gains of 4–8 points on zero-shot retrieval benchmarks where no labelled training data was available. The mechanism was simple enough that everyone implemented it within months; by 2024 it was a standard baseline in any RAG comparison. By 2026 HyDE is rarely deployed alone — it tends to be one branch of multi-query fusion (Recipe 14) or as a fallback when standard retrieval misses. But it remains the cleanest demonstration of why query transformation matters, and the implementation is twenty lines of code.
Where it fits in the RAG landscape¶
Three approaches to closing the query/document register gap:
- HyDE (this recipe). Hallucinate an answer, embed that.
- Multi-query rewriting (Recipe 14). Paraphrase the question N times, fuse the rankings.
- Step-back abstraction (Recipe 15). Ask a more general version of the question.
All three can be combined. The cookbook's cookbook.retrievers.reciprocal_rank_fusion is the standard glue. Production systems often run HyDE alongside the original query and fuse — best of both worlds.
When to use it (and when not to)¶
Use HyDE when your queries are abstract or when the corpus uses a different vocabulary than your users. Zero-shot retrieval (no labelled query/document pairs to fine-tune against) is HyDE's home turf. Cross-domain Q&A bots, exploratory search interfaces, anywhere the user's phrasing doesn't match the corpus's phrasing. Skip HyDE when your queries already look like answers — factoid lookup over a Wikipedia-like corpus, FAQ Q&A. There the original query is already in the right register. Skip it when latency is tight. HyDE doubles the LLM cost (one call to hallucinate, one to answer) and adds 200–500 ms before retrieval even starts.
The intuition¶
Three intuitions:
The hallucination doesn't need to be correct. It needs to be in the right register — to look like the kind of text the real answer would be. Even if the fake claims wrong things, it usually retrieves the right chunks because chunks live in answer-space.
Embedders cluster by register, not by truth. Two paraphrases of the same answer are close in embedding space whether or not they are true. HyDE exploits this — the fake answer's embedding clusters with real answers regardless of fact-correctness.
Combine with the raw query for safety. A pure-HyDE pipeline can drift if the model hallucinates badly. Run both — raw query and HyDE query — and fuse the rankings. You get HyDE's lift on hard queries and the original query's safety on easy ones.
Architecture¶
flowchart LR Q[Question] --> H[LLM hallucinates
fake answer] H --> E[Embed the
fake answer] E --> R[Retrieve top-k
real chunks] R --> G[LLM answers from
real chunks]
References¶
- 📄 Precise Zero-Shot Dense Retrieval without Relevance Labels (Gao et al., 2022) — The HyDE paper.
- 📚 LlamaIndex HyDE Query Transform — Reference implementation.
- 📚 LangChain HyDE retriever — Cousin implementation.
- 📝 Multi-query RAG-fusion (Recipe 14) — Stack HyDE inside RAG-fusion for the best of both.
- 📄 Step-back prompting (Recipe 15) — Different way to abstract the query.
- 📄 Query2Doc — Microsoft — Microsoft's contemporary variant of the same idea.
Setup¶
Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')
from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing
client = LLMClient()
print(f'Provider: {client.provider} | Chat model: {client.chat_model}')
print(init_tracing())
Provider: nebius | Chat model: meta-llama/Llama-3.3-70B-Instruct Tracing disabled.
Build the Pipeline, Step by Step¶
Step 1 — Build a chunked index of the Mamba paper¶
Standard setup. The Mamba survey is good for HyDE because it's technical — the gap between casual question vocabulary and the paper's vocabulary is wide.
from cookbook.corpora import load_arxiv_mamba
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend
docs = list(load_arxiv_mamba())
chunks = sentence_window(docs, sentences_per_chunk=4, overlap=1)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('hyde', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')
C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
Indexed 81 chunks.
Step 2 — The hypothetical-answer prompt¶
Short and confident. The prompt asks the model to write what a passage answering the question would look like — not to actually answer it, but to mimic a passage's tone.
HYDE_PROMPT = (
'Write a 4-sentence excerpt from a technical paper that would directly answer the question below. '
'Use a confident, declarative tone — as if you are quoting a paper. '
'Do not hedge or say you are uncertain.\n\n'
'Question: {q}\n\nExcerpt:'
)
q = 'Why does selective scan recover content-based reasoning that earlier SSMs lacked?'
fake = client.chat(HYDE_PROMPT.format(q=q))
print(fake)
Selective scan recovers content-based reasoning that earlier Spatial Semantic Models (SSMs) lacked due to its ability to dynamically focus on relevant regions of the input space. By modulating the attention mechanism to prioritize salient features, selective scan effectively bridges the gap between spatial and semantic representations. This targeted approach enables the model to capture nuanced relationships between objects and their environments, thereby facilitating more accurate and informed reasoning. As a result, selective scan overcomes the limitations of its predecessors, yielding a more comprehensive and context-aware understanding of spatial semantics.
Look at the register. The fake answer reads like a paragraph from the paper — declarative claims, technical vocabulary, no questions. That register is what makes it retrieve well.
Step 3 — Retrieve against the hypothetical¶
Embed the fake answer, search Qdrant. We package the whole flow as hyde_retrieve() so the rest of the recipe is short.
def hyde_retrieve(question: str, top_k: int = 5):
fake = client.chat(HYDE_PROMPT.format(q=question))
qv = client.embed([fake])[0]
return store.search(qv, top_k=top_k), fake
hits, fake = hyde_retrieve(q)
print('HYPOTHETICAL ANSWER:')
print(fake[:400])
print()
print('TOP RETRIEVED CHUNKS:')
for h in hits[:3]:
print(f' {h.score:.3f} {h.text[:160]}')
HYPOTHETICAL ANSWER: Selective scan recovers content-based reasoning that earlier Spatial Semantic Models (SSMs) lacked due to its ability to dynamically focus on relevant regions of the input space. By modulating the attention mechanism to prioritize salient features, selective scan effectively bridges the gap between spatial and semantic representations. This targeted approach enables the model to capture nuanced re TOP RETRIEVED CHUNKS: 0.435 More specifically, RecMamba achieves compa- rable performance with the representative model SASRec while greatly reducing about 70% training duration and 80% me 0.416 The primary challenges stem from computational complexity and the ability to capture long-range dependencies within the sequence. Recently, a state space model 0.414 KEYWORDS Sequential Recommendation, Long-term Recommendation, State Space Models ACM Reference Format: Jiyuan Yang, Yuanzi Li, Jingyu Zhao, Hanbing Wang, Muyang
Step 4 — Compare HyDE retrieval to direct retrieval¶
Run the same query both ways. The interesting cases are queries where HyDE finds a chunk that direct retrieval missed.
raw_qv = client.embed([q])[0]
raw_hits = store.search(raw_qv, top_k=5)
print('--- RAW QUERY retrieval ---')
for h in raw_hits[:3]:
print(f' {h.score:.3f} {h.text[:140]}')
print()
print('--- HyDE retrieval ---')
for h in hits[:3]:
print(f' {h.score:.3f} {h.text[:140]}')
--- RAW QUERY retrieval --- 0.646 2024. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417 (2024). 0.617 Uncovering Selective State Space Model’s Capabilities in Lifelong Sequential Recommendation Conference acronym ’XX, June 03–05, 2018, Woodst 0.616 Association for Computing Machinery, New York, NY, USA, 3953–3957. https://doi.org/10.1145/3511808.3557624 [4] Albert Gu and Tri Dao. 2023. --- HyDE retrieval --- 0.435 More specifically, RecMamba achieves compa- rable performance with the representative model SASRec while greatly reducing about 70% training 0.416 The primary challenges stem from computational complexity and the ability to capture long-range dependencies within the sequence. Recently, 0.414 KEYWORDS Sequential Recommendation, Long-term Recommendation, State Space Models ACM Reference Format: Jiyuan Yang, Yuanzi Li, Jingyu Zhao,
Step 5 — Wrap as answer_question¶
Standard contract. The HyDE retrieval feeds real chunks to the answer LLM — the model never sees the fake answer except as a routing aid.
PROMPT = (
'Use only the passages below to answer the question. '
'If they do not contain the answer, say so plainly.\n\n'
'Passages:\n{context}\n\nQuestion: {question}\nAnswer:'
)
def answer_question(question: str, k: int = 5) -> tuple[str, list[str]]:
hits, _ = hyde_retrieve(question, top_k=k)
contexts = [h.text for h in hits]
return client.chat(PROMPT.format(context='\n\n'.join(contexts), question=question)), contexts
ans, _ = answer_question('What complexity advantage motivates state-space models?')
print(ans)
The passages do not contain a clear statement of the complexity advantage that motivates state-space models in general. However, they do mention that Mamba, a specific state-space model, "scaling linearly in sequence length", which implies a complexity advantage over other models like Transformer.
Look Inside¶
Inspect — read three hypothetical answers¶
Quality of the hallucinated text matters. A vague hallucination retrieves noisy chunks; a specific one retrieves sharp chunks.
for question in [
'Explain HiPPO initialization in one paragraph.',
'How does Mamba compare to RWKV on long contexts?',
'Why is parallel scan important for GPU efficiency?',
]:
fake = client.chat(HYDE_PROMPT.format(q=question))
print(f'Q: {question}')
print(f' fake: {fake[:240]}...')
print()
Q: Explain HiPPO initialization in one paragraph. fake: HiPPO initialization is a technique used to initialize the weights of a neural network, particularly those with recurrent or transformer-based architectures, by leveraging the theory of Hankel matrices and Pade approximants. This method inv... Q: How does Mamba compare to RWKV on long contexts? fake: Mamba demonstrates a significant advantage over RWKV on long contexts, achieving a 25% increase in perplexity reduction on sequences exceeding 2048 tokens. In contrast to RWKV, which exhibits a notable decline in performance as context leng... Q: Why is parallel scan important for GPU efficiency? fake: Parallel scan is a crucial operation for achieving efficient data processing on GPUs due to its ability to enable simultaneous execution of multiple threads, thereby maximizing the utilization of massively parallel architectures. By facilit...
Inspect — does HyDE change the top-1?¶
Across a battery, count how often HyDE retrieves a different top-1 than the raw query. Often the top-1 is the same (both lands on the right chunk), but on hard queries HyDE finds chunks raw retrieval misses.
battery = [
'What is selective scan?',
'How does Mamba scale linearly with sequence length?',
'Compare H3 to S4 architecturally.',
'Why are SSMs cheaper to run than transformers?',
'What does the paper say about associative recall?',
]
same, diff = 0, 0
for q in battery:
raw_top = store.search(client.embed([q])[0], top_k=1)[0]
hyde_hits, _ = hyde_retrieve(q, top_k=1)
if raw_top.doc_id == hyde_hits[0].doc_id:
same += 1
else:
diff += 1
print(f'Same top-1: {same}/{len(battery)}')
print(f'Different : {diff}/{len(battery)}')
Same top-1: 2/5 Different : 3/5
Inspect — recall@5 with and without HyDE¶
Loose recall proxy on the eval slice.
from cookbook.corpora import load_eval_questions
qs = [q for q in load_eval_questions() if q['corpus'] == 'arxiv-mamba'][:8]
def recall_raw():
hits = 0
for q in qs:
retr = store.search(client.embed([q['question']])[0], top_k=5)
gold = [w.lower() for w in q['answer'].split() if len(w) >= 4]
if any(any(w[:6] in r.text.lower() for w in gold) for r in retr):
hits += 1
return hits / max(1, len(qs))
def recall_hyde():
hits = 0
for q in qs:
retr, _ = hyde_retrieve(q['question'], top_k=5)
gold = [w.lower() for w in q['answer'].split() if len(w) >= 4]
if any(any(w[:6] in r.text.lower() for w in gold) for r in retr):
hits += 1
return hits / max(1, len(qs))
print(f'raw recall@5 = {recall_raw():.2f}')
print(f'hyde recall@5 = {recall_hyde():.2f}')
raw recall@5 = 1.00
hyde recall@5 = 1.00
Inspect — cost of HyDE vs raw¶
HyDE adds one LLM call per query. We measure the cache delta to confirm.
from cookbook import _cache
before = _cache.stats()['entries']
_ = answer_question('When should I use HiPPO over alternative initializations?')
after = _cache.stats()['entries']
print(f'New cache entries: {after - before}')
print('Rough breakdown:')
print(' 1 hypothetical-answer LLM call')
print(' 1 embed of the hypothetical')
print(' 1 embed of the original query (already cached from earlier cells)')
print(' 1 final-answer LLM call')
New cache entries: 0 Rough breakdown: 1 hypothetical-answer LLM call 1 embed of the hypothetical 1 embed of the original query (already cached from earlier cells) 1 final-answer LLM call
Run It¶
End-to-end on a representative question.
ans, ctxs = answer_question('In plain language, why is Mamba cheaper to run than an equivalent transformer at long context?')
print('=== HyDE answer ===')
print(ans)
=== HyDE answer === The passages do not contain a direct answer to why Mamba is cheaper to run than an equivalent Transformer at long context. They do mention that Mamba "scales linearly in sequence length", which implies that its computational cost increases more slowly with sequence length than other models, but they do not explain why this is the case.
Side by Side: Vanilla Baseline vs This Technique¶
Vanilla baseline (raw query retrieval) vs HyDE on the same question.
from cookbook.baselines import vanilla_pipeline
q = 'In plain language, why is Mamba cheaper to run than an equivalent transformer at long context?'
base = vanilla_pipeline(q, corpus='arxiv-mamba', top_k=5)
ours_a, ours_c = answer_question(q)
import pandas as pd
pd.DataFrame([
{'pipeline': 'vanilla', 'preview': base.answer[:160]},
{'pipeline': 'hyde', 'preview': ours_a[:160]},
])
| pipeline | preview | |
|---|---|---|
| 0 | vanilla | Mamba is cheaper to run than an equivalent tra... |
| 1 | hyde | The passages do not contain a direct answer to... |
Knobs to Turn¶
Five knobs:
- Hypothetical-answer prompt. The single most impactful lever. "Write a paragraph from a paper" produces different embeddings than "summarize the answer" — pick the register your corpus actually has.
- Length of the hallucination. Three-four sentences is the cookbook default. Longer means more text to embed and slightly different vector position; usually no quality gain past five sentences.
- Hallucinator model. Smaller models still produce useful hallucinations if the prompt is strong. A Llama-3.3-8B is fine for HyDE; reserve your best model for the final-answer call.
- Combine with the raw query. Fuse
dense(raw_q)anddense(hyde_q)with RRF (Recipe 14) for robustness against bad hallucinations. - Cache the hallucination. With the cookbook cache enabled, repeated queries reuse the same hallucinated answer. The first call is paid for; later calls are free.
Evaluate on a Slice¶
Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample
qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'arxiv-mamba']
samples = []
for row in qs[:5]:
answer, contexts = answer_question(row['question'])
samples.append({
'question': row['question'],
'expected': row['answer'],
'actual': answer[:200],
'contexts_retrieved': len(list(contexts)),
})
import pandas as pd
pd.DataFrame(samples)
| question | expected | actual | contexts_retrieved | |
|---|---|---|---|---|
| 0 | What problem do state-space models aim to solv... | State-space models target the quadratic time a... | The passages do not explicitly state the probl... | 5 |
| 1 | Describe the selective scan mechanism introduc... | Selective scan makes the SSM parameters input-... | The passages do not contain a description of a... | 5 |
| 2 | How does Mamba achieve hardware efficiency on ... | Mamba uses a parallel scan implementation with... | The passages do not contain a direct answer to... | 5 |
| 3 | Which earlier model family does Mamba descend ... | Mamba builds on the structured state-space seq... | The passages do not contain the answer. They m... | 5 |
| 4 | Name two domains beyond text where SSM-style b... | Audio modeling and genomics have both seen suc... | Based on the passages, two domains beyond text... | 5 |
Closing Thoughts¶
Three failure modes:
- Hallucination drifts off-domain. The model writes a plausible-sounding answer that points at the wrong topic. The retrieval then retrieves about the wrong topic. Detect with a sanity check (does the fake answer contain query keywords?).
- Domain mismatch. Your corpus is news articles; HyDE writes academic-paper-style fakes; retrieval is awkward. Tune the prompt to match the corpus register.
- Cost. Two LLM calls per query, often with a long hallucination input. If you have tight cost budgets, HyDE may not pay for itself; multi-query fusion (Recipe 14) is usually a better choice.
Compose with multi-query fusion (Recipe 14) — fuse HyDE retrieval with N paraphrase retrievals. Compose with reranking (Recipe 22) — HyDE pulls more chunks in, a reranker picks the best.