Speculative RAG — Drafter and Verifier in Parallel¶
What problem does this solve?¶
Standard RAG runs one expensive LLM call per query. The model receives k chunks, reasons over them, and produces an answer. The reasoning is sequential, the cost is linear in answer length, and the latency is whatever the model takes — usually 1–3 seconds. Speculative RAG borrows the idea from speculative decoding: run several cheap drafter models in parallel, then have one expensive verifier pick the best draft. Each drafter sees a different retrieval subset, so their drafts are diverse. The verifier reads them together and picks. Latency drops because drafting is parallel; quality often improves because the verifier sees diverse evidence.
Where it came from¶
Speculative RAG was published at ICLR 2025 (Wang et al.). The paper demonstrated 50% latency reductions at comparable or better quality on TriviaQA, HotpotQA, and PubMedQA. The technique borrowed from speculative decoding (Stern et al., 2018; Leviathan et al., 2023) which uses a small drafter to predict tokens that a large verifier accepts or rejects, then lifted that pattern to the document-retrieval level. By 2026 the pattern has spread to anything where latency budgets are tight. Customer-support assistants, real-time chat agents, and voice interfaces all benefit from the parallel-draft structure. The technique pairs especially well with prompt caching — drafter prompts are very similar across queries, so cache hit rates are high.
Where it fits in the RAG landscape¶
Three approaches to fast RAG you should know:
- Caching (Recipe 2 baseline). Cache embeddings and LLM responses for repeat queries. Free first-line defence.
- Speculative RAG (this recipe). Parallel cheap drafts, single verify. Cuts latency by parallelising the drafting step.
- Streaming generation. Start emitting tokens before all chunks arrive. Reduces perceived latency.
Speculative composes with caching and streaming — they attack different parts of the latency budget. A production stack often runs all three.
When to use it (and when not to)¶
Use speculative when you need fast RAG with comparable quality. The technique shines when the corpus has multiple plausible retrieval subsets — i.e., when reranking improves things and the right top-k isn't obvious. Skip it when you have only one obvious retrieval set. With k=3 and only 4 viable chunks, you can't run diverse drafters and the parallel structure wastes budget. Skip it when budget matters more than latency. Speculative makes N+1 LLM calls per query instead of 1, which is a meaningful multiplier at high volume.
The intuition¶
Three intuitions to carry:
Parallel drafts give the verifier choices. With three drafters seeing different chunk subsets, the verifier sees three candidate answers and picks. Without speculative, the LLM would have to produce one answer and accept it.
The verifier only picks — it doesn't generate. This keeps the expensive call short. The verifier prompt is just "which of these is best?".
Latency = max(drafters) + verify, not sum. If drafters take 800 ms each and the verifier takes 200 ms, total wall time is 1 second — not 3 seconds.
Drafter diversity matters. All drafters seeing the same chunks would produce the same draft and waste the parallel structure. The cookbook splits the candidate pool into N disjoint subsets, one per drafter, so each drafter sees genuinely different evidence and produces a genuinely different draft.
Architecture¶
flowchart TB Q[Query] --> R[Retrieve top-k*N] R --> S1[Drafter 1
subset 1] R --> S2[Drafter 2
subset 2] R --> S3[Drafter 3
subset 3] S1 --> V[Verifier:
pick best draft] S2 --> V S3 --> V V --> A[Final answer]
References¶
- 📄 Speculative RAG — Enhancing Retrieval Augmented Generation through Drafting (Wang et al., 2025) — The ICLR 2025 paper.
- 📄 Speculative Decoding (Leviathan et al., 2023) — The decoding-time precursor that inspired Speculative RAG.
- 📚 LlamaIndex Speculative RAG reference — Reference implementation.
- 📄 Self-RAG (Recipe 24) — Cousin technique — sequential reflection rather than parallel drafting.
- 📚 OpenAI prompt caching — Composes well — the verifier sees similar inputs across queries.
- 📚 Streaming generation cookbook (LiteLLM) — Combine with speculative for additional latency wins.
Setup¶
Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.
In [1]:
Copied!
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')
from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing
client = LLMClient()
print(f'Provider: {client.provider} | Chat model: {client.chat_model}')
print(init_tracing())
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')
from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing
client = LLMClient()
print(f'Provider: {client.provider} | Chat model: {client.chat_model}')
print(init_tracing())
Provider: nebius | Chat model: meta-llama/Llama-3.3-70B-Instruct Tracing disabled.
Build the Pipeline, Step by Step¶
Step 1 — Build the index¶
Standard Mamba paper setup.
In [2]:
Copied!
from cookbook.corpora import load_arxiv_mamba
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend
docs = list(load_arxiv_mamba())
chunks = sentence_window(docs, sentences_per_chunk=4)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('spec', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')
from cookbook.corpora import load_arxiv_mamba
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend
docs = list(load_arxiv_mamba())
chunks = sentence_window(docs, sentences_per_chunk=4)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('spec', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')
C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
Indexed 81 chunks.
Step 2 — Configure drafter and verifier¶
Drafter is a small fast model; verifier is the default chat model. In production you'd pick a smaller drafter (Qwen 7B) and reserve frontier for the verifier.
In [3]:
Copied!
drafter = client
verifier = client
print(f'Drafter model: {drafter.chat_model}')
print(f'Verifier model: {verifier.chat_model}')
drafter = client
verifier = client
print(f'Drafter model: {drafter.chat_model}')
print(f'Verifier model: {verifier.chat_model}')
Drafter model: meta-llama/Llama-3.3-70B-Instruct Verifier model: meta-llama/Llama-3.3-70B-Instruct
Step 3 — Split candidates into drafter subsets¶
Retrieve k*N candidates, partition into N disjoint subsets. Each drafter sees one subset.
In [4]:
Copied!
import math
def split_subsets(hits, n_drafters: int):
return [hits[i::n_drafters] for i in range(n_drafters)]
q = 'How does the parallel scan algorithm exploit GPU memory hierarchies?'
qv = client.embed([q])[0]
pool = store.search(qv, top_k=15)
subsets = split_subsets(pool, n_drafters=3)
for i, s in enumerate(subsets):
print(f'Subset {i+1}: {len(s)} chunks; first chunk preview: {s[0].text[:60]}...')
import math
def split_subsets(hits, n_drafters: int):
return [hits[i::n_drafters] for i in range(n_drafters)]
q = 'How does the parallel scan algorithm exploit GPU memory hierarchies?'
qv = client.embed([q])[0]
pool = store.search(qv, top_k=15)
subsets = split_subsets(pool, n_drafters=3)
for i, s in enumerate(subsets):
print(f'Subset {i+1}: {len(s)} chunks; first chunk preview: {s[0].text[:60]}...')
Subset 1: 5 chunks; first chunk preview: This advan- tage becomes more pronounced as the sequence len... Subset 2: 5 chunks; first chunk preview: We can observe that RecMamba notably reduces GPU memory foot... Subset 3: 5 chunks; first chunk preview: More specifically, we delve into crucial efficiency metrics,...
Step 4 — Draft in parallel¶
We run N drafters concurrently. Each produces a candidate answer.
In [5]:
Copied!
from concurrent.futures import ThreadPoolExecutor
DRAFT_PROMPT = (
'Use the passages below to answer the question concisely.\n'
'Passages:\n{ctx}\nQuestion: {q}\nAnswer:'
)
def draft_one(s):
ctx = '\n\n'.join(h.text for h in s)
return drafter.chat(DRAFT_PROMPT.format(ctx=ctx, q=q))
with ThreadPoolExecutor(max_workers=len(subsets)) as ex:
drafts = list(ex.map(draft_one, subsets))
for i, d in enumerate(drafts):
print(f'Draft {i+1}: {d[:200]}...')
print()
from concurrent.futures import ThreadPoolExecutor
DRAFT_PROMPT = (
'Use the passages below to answer the question concisely.\n'
'Passages:\n{ctx}\nQuestion: {q}\nAnswer:'
)
def draft_one(s):
ctx = '\n\n'.join(h.text for h in s)
return drafter.chat(DRAFT_PROMPT.format(ctx=ctx, q=q))
with ThreadPoolExecutor(max_workers=len(subsets)) as ex:
drafts = list(ex.map(draft_one, subsets))
for i, d in enumerate(drafts):
print(f'Draft {i+1}: {d[:200]}...')
print()
Draft 1: The passage does not mention how the parallel scan algorithm exploits GPU memory hierarchies. It discusses the efficiency of RecMamba, a sequence recommendation framework, but does not provide informa... Draft 2: The passage does not explicitly explain how the parallel scan algorithm exploits GPU memory hierarchies. It only mentions that RecMamba "utilizes a parallel algorithm optimized for hardware in recurre... Draft 3: The passage does not explicitly explain how the parallel scan algorithm exploits GPU memory hierarchies. However, it mentions that Mamba "utilizes a parallel algorithm optimized for hardware in recurr...
Step 5 — Verify and pick¶
The verifier reads all N drafts and picks the best by index. We constrain output to just the index for cheap parsing.
In [6]:
Copied!
rendered = '\n\n'.join(f'[{i}] {d}' for i, d in enumerate(drafts))
pick_prompt = (
'Pick the best draft index given the question. Reply with just the index.\n'
+ rendered + f'\nQuestion: {q}'
)
pick_raw = verifier.chat(pick_prompt)
import re
m = re.search(r'\d+', pick_raw)
idx = int(m.group()) if m else 0
idx = max(0, min(idx, len(drafts) - 1))
final = drafts[idx]
print(f'Verifier picked draft {idx}.')
print()
print(final)
rendered = '\n\n'.join(f'[{i}] {d}' for i, d in enumerate(drafts))
pick_prompt = (
'Pick the best draft index given the question. Reply with just the index.\n'
+ rendered + f'\nQuestion: {q}'
)
pick_raw = verifier.chat(pick_prompt)
import re
m = re.search(r'\d+', pick_raw)
idx = int(m.group()) if m else 0
idx = max(0, min(idx, len(drafts) - 1))
final = drafts[idx]
print(f'Verifier picked draft {idx}.')
print()
print(final)
Verifier picked draft 2. The passage does not explicitly explain how the parallel scan algorithm exploits GPU memory hierarchies. However, it mentions that Mamba "utilizes a parallel algorithm optimized for hardware in recurrent mode", which suggests that the algorithm is designed to take advantage of the hardware architecture, potentially including GPU memory hierarchies, to enable effective sequence modeling.
Step 6 — Wrap as answer_question¶
Cookbook contract.
In [7]:
Copied!
def answer_question(question: str, n_drafters: int = 3, k: int = 5) -> tuple[str, list[str]]:
qv = client.embed([question])[0]
pool = store.search(qv, top_k=k * n_drafters)
subsets = split_subsets(pool, n_drafters)
def draft_one_inner(s):
ctx = '\n\n'.join(h.text for h in s)
return drafter.chat(DRAFT_PROMPT.format(ctx=ctx, q=question))
with ThreadPoolExecutor(max_workers=n_drafters) as ex:
drafts = list(ex.map(draft_one_inner, subsets))
rendered = '\n\n'.join(f'[{i}] {d}' for i, d in enumerate(drafts))
pick_raw = verifier.chat(
'Pick the best draft index given the question. Reply with just the index.\n'
+ rendered + f'\nQuestion: {question}'
)
m = re.search(r'\d+', pick_raw)
idx = max(0, min(int(m.group()) if m else 0, n_drafters - 1))
return drafts[idx], [h.text for h in subsets[idx]]
ans, _ = answer_question('Why is HiPPO initialization useful?')
print(ans)
def answer_question(question: str, n_drafters: int = 3, k: int = 5) -> tuple[str, list[str]]:
qv = client.embed([question])[0]
pool = store.search(qv, top_k=k * n_drafters)
subsets = split_subsets(pool, n_drafters)
def draft_one_inner(s):
ctx = '\n\n'.join(h.text for h in s)
return drafter.chat(DRAFT_PROMPT.format(ctx=ctx, q=question))
with ThreadPoolExecutor(max_workers=n_drafters) as ex:
drafts = list(ex.map(draft_one_inner, subsets))
rendered = '\n\n'.join(f'[{i}] {d}' for i, d in enumerate(drafts))
pick_raw = verifier.chat(
'Pick the best draft index given the question. Reply with just the index.\n'
+ rendered + f'\nQuestion: {question}'
)
m = re.search(r'\d+', pick_raw)
idx = max(0, min(int(m.group()) if m else 0, n_drafters - 1))
return drafts[idx], [h.text for h in subsets[idx]]
ans, _ = answer_question('Why is HiPPO initialization useful?')
print(ans)
There is no mention of HiPPO initialization in the provided passages, so it is not possible to answer the question based on the given text.
Look Inside¶
Inspect — how often does the verifier pick non-default drafts?¶
If the verifier always picks draft 0, speculative isn't doing work — drafts 1 and 2 are wasted.
In [8]:
Copied!
from collections import Counter
picks = Counter()
for q in [
'What is selective scan?',
'How does Mamba differ from S4?',
'Why is parallel scan important?',
'When should I use a state-space model?',
]:
qv = client.embed([q])[0]
pool = store.search(qv, top_k=15)
subsets = split_subsets(pool, n_drafters=3)
def d(s):
ctx = '\n\n'.join(h.text for h in s)
return drafter.chat(DRAFT_PROMPT.format(ctx=ctx, q=q))
with ThreadPoolExecutor(max_workers=3) as ex:
ds = list(ex.map(d, subsets))
rendered = '\n\n'.join(f'[{i}] {x}' for i, x in enumerate(ds))
p = verifier.chat(
'Pick the best draft index. Reply with just the index.\n'
+ rendered + f'\nQuestion: {q}'
)
m = re.search(r'\d+', p)
picks[int(m.group()) if m else 0] += 1
print(f'Picks by index: {dict(picks)}')
from collections import Counter
picks = Counter()
for q in [
'What is selective scan?',
'How does Mamba differ from S4?',
'Why is parallel scan important?',
'When should I use a state-space model?',
]:
qv = client.embed([q])[0]
pool = store.search(qv, top_k=15)
subsets = split_subsets(pool, n_drafters=3)
def d(s):
ctx = '\n\n'.join(h.text for h in s)
return drafter.chat(DRAFT_PROMPT.format(ctx=ctx, q=q))
with ThreadPoolExecutor(max_workers=3) as ex:
ds = list(ex.map(d, subsets))
rendered = '\n\n'.join(f'[{i}] {x}' for i, x in enumerate(ds))
p = verifier.chat(
'Pick the best draft index. Reply with just the index.\n'
+ rendered + f'\nQuestion: {q}'
)
m = re.search(r'\d+', p)
picks[int(m.group()) if m else 0] += 1
print(f'Picks by index: {dict(picks)}')
Picks by index: {1: 2, 0: 1, 2: 1}
Inspect — draft diversity¶
Drafters seeing different chunks should produce different drafts. We measure character-level overlap to confirm.
In [9]:
Copied!
from difflib import SequenceMatcher
q = 'How does Mamba scale linearly with sequence length?'
qv = client.embed([q])[0]
pool = store.search(qv, top_k=12)
subsets = split_subsets(pool, n_drafters=3)
def df(s):
ctx = '\n\n'.join(h.text for h in s)
return drafter.chat(DRAFT_PROMPT.format(ctx=ctx, q=q))
with ThreadPoolExecutor(max_workers=3) as ex:
drafts = list(ex.map(df, subsets))
for i, d in enumerate(drafts):
print(f'Draft {i+1} length: {len(d)} chars')
for i in range(len(drafts)):
for j in range(i+1, len(drafts)):
ratio = SequenceMatcher(None, drafts[i], drafts[j]).ratio()
print(f' similarity draft {i+1} vs {j+1}: {ratio:.2f}')
from difflib import SequenceMatcher
q = 'How does Mamba scale linearly with sequence length?'
qv = client.embed([q])[0]
pool = store.search(qv, top_k=12)
subsets = split_subsets(pool, n_drafters=3)
def df(s):
ctx = '\n\n'.join(h.text for h in s)
return drafter.chat(DRAFT_PROMPT.format(ctx=ctx, q=q))
with ThreadPoolExecutor(max_workers=3) as ex:
drafts = list(ex.map(df, subsets))
for i, d in enumerate(drafts):
print(f'Draft {i+1} length: {len(d)} chars')
for i in range(len(drafts)):
for j in range(i+1, len(drafts)):
ratio = SequenceMatcher(None, drafts[i], drafts[j]).ratio()
print(f' similarity draft {i+1} vs {j+1}: {ratio:.2f}')
Draft 1 length: 247 chars Draft 2 length: 209 chars Draft 3 length: 237 chars similarity draft 1 vs 2: 0.23 similarity draft 1 vs 3: 0.22 similarity draft 2 vs 3: 0.21
Inspect — latency breakdown¶
Drafts run in parallel; verify runs once. The bottleneck is the slowest drafter plus the verifier.
In [10]:
Copied!
import time
q = 'What is selective scan?'
qv = client.embed([q])[0]
pool = store.search(qv, top_k=15)
subsets = split_subsets(pool, n_drafters=3)
t0 = time.perf_counter()
def d(s):
ctx = '\n\n'.join(h.text for h in s)
return drafter.chat(DRAFT_PROMPT.format(ctx=ctx, q=q))
with ThreadPoolExecutor(max_workers=3) as ex:
drafts = list(ex.map(d, subsets))
draft_ms = (time.perf_counter() - t0) * 1000
t0 = time.perf_counter()
rendered = '\n\n'.join(f'[{i}] {x}' for i, x in enumerate(drafts))
_ = verifier.chat('Pick the best draft index. Reply with just the index.\n' + rendered + f'\nQuestion: {q}')
verify_ms = (time.perf_counter() - t0) * 1000
print(f'Parallel draft (wall-clock): {draft_ms:.1f} ms')
print(f'Verify (single call): {verify_ms:.1f} ms')
print(f'Total: {draft_ms + verify_ms:.1f} ms')
import time
q = 'What is selective scan?'
qv = client.embed([q])[0]
pool = store.search(qv, top_k=15)
subsets = split_subsets(pool, n_drafters=3)
t0 = time.perf_counter()
def d(s):
ctx = '\n\n'.join(h.text for h in s)
return drafter.chat(DRAFT_PROMPT.format(ctx=ctx, q=q))
with ThreadPoolExecutor(max_workers=3) as ex:
drafts = list(ex.map(d, subsets))
draft_ms = (time.perf_counter() - t0) * 1000
t0 = time.perf_counter()
rendered = '\n\n'.join(f'[{i}] {x}' for i, x in enumerate(drafts))
_ = verifier.chat('Pick the best draft index. Reply with just the index.\n' + rendered + f'\nQuestion: {q}')
verify_ms = (time.perf_counter() - t0) * 1000
print(f'Parallel draft (wall-clock): {draft_ms:.1f} ms')
print(f'Verify (single call): {verify_ms:.1f} ms')
print(f'Total: {draft_ms + verify_ms:.1f} ms')
Parallel draft (wall-clock): 2.0 ms Verify (single call): 0.3 ms Total: 2.4 ms
Inspect — cost¶
Speculative makes N+1 LLM calls per query. Track them.
In [11]:
Copied!
from cookbook import _cache
before = _cache.stats()['entries']
_ = answer_question('What is HiPPO initialization?')
after = _cache.stats()['entries']
print(f'New cache entries: {after - before}')
print('Rough breakdown for n_drafters=3:')
print(' 3 drafter LLM calls')
print(' 1 verifier LLM call')
print(' 1 query embed')
from cookbook import _cache
before = _cache.stats()['entries']
_ = answer_question('What is HiPPO initialization?')
after = _cache.stats()['entries']
print(f'New cache entries: {after - before}')
print('Rough breakdown for n_drafters=3:')
print(' 3 drafter LLM calls')
print(' 1 verifier LLM call')
print(' 1 query embed')
New cache entries: 0 Rough breakdown for n_drafters=3: 3 drafter LLM calls 1 verifier LLM call 1 query embed
Run It¶
End-to-end on a representative question.
In [12]:
Copied!
ans, _ = answer_question('What does Mamba claim about long-context efficiency, and what trade-off makes it possible?')
print('=== Speculative answer ===')
print(ans)
ans, _ = answer_question('What does Mamba claim about long-context efficiency, and what trade-off makes it possible?')
print('=== Speculative answer ===')
print(ans)
=== Speculative answer === Mamba claims to scale linearly in sequence length, achieving the modeling power of Transformer while improving long-context efficiency. This is made possible by a trade-off between modeling power and computational complexity, specifically through its selective mechanism and parallel algorithm optimized for hardware, which reduces training duration by 70% and memory costs by 80%.
Side by Side: Vanilla Baseline vs This Technique¶
Vanilla (single LLM call) vs speculative (N drafters + 1 verifier).
In [13]:
Copied!
from cookbook.baselines import vanilla_pipeline
q = 'What does Mamba claim about long-context efficiency?'
base = vanilla_pipeline(q, corpus='arxiv-mamba', top_k=5)
ours_a, _ = answer_question(q)
import pandas as pd
pd.DataFrame([
{'pipeline': 'vanilla', 'preview': base.answer[:160]},
{'pipeline': 'speculative', 'preview': ours_a[:160]},
])
from cookbook.baselines import vanilla_pipeline
q = 'What does Mamba claim about long-context efficiency?'
base = vanilla_pipeline(q, corpus='arxiv-mamba', top_k=5)
ours_a, _ = answer_question(q)
import pandas as pd
pd.DataFrame([
{'pipeline': 'vanilla', 'preview': base.answer[:160]},
{'pipeline': 'speculative', 'preview': ours_a[:160]},
])
Out[13]:
| pipeline | preview | |
|---|---|---|
| 0 | vanilla | The passages do not explicitly state what Mamb... |
| 1 | speculative | Mamba (specifically RecMamba) claims to have a... |
Knobs to Turn¶
Six knobs in priority order:
n_drafters. 3 is the sweet spot. Higher pays diminishing returns and inflates cost without quality gain.- Drafter model. Use a small fast model. A frontier drafter wastes budget without quality gain.
- Verifier model. A mid-tier model is enough. The verifier only picks, doesn't generate.
- Subset size. k chunks per drafter × N drafters = pool size. We use k=5 per drafter, 3 drafters → 15 total.
- Streaming verifier. When the verifier is streaming, you can return the picked draft as soon as the index token arrives.
- Parallel-thread cap. Cap the executor's max_workers to your provider's concurrency limit to avoid rate-limit errors.
Evaluate on a Slice¶
Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.
In [14]:
Copied!
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample
qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'arxiv-mamba']
samples = []
for row in qs[:5]:
answer, contexts = answer_question(row['question'])
samples.append({
'question': row['question'],
'expected': row['answer'],
'actual': answer[:200],
'contexts_retrieved': len(list(contexts)),
})
import pandas as pd
pd.DataFrame(samples)
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample
qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'arxiv-mamba']
samples = []
for row in qs[:5]:
answer, contexts = answer_question(row['question'])
samples.append({
'question': row['question'],
'expected': row['answer'],
'actual': answer[:200],
'contexts_retrieved': len(list(contexts)),
})
import pandas as pd
pd.DataFrame(samples)
Out[14]:
| question | expected | actual | contexts_retrieved | |
|---|---|---|---|---|
| 0 | What problem do state-space models aim to solv... | State-space models target the quadratic time a... | State-space models, such as Mamba, aim to solv... | 5 |
| 1 | Describe the selective scan mechanism introduc... | Selective scan makes the SSM parameters input-... | The passages do not describe the selective sca... | 5 |
| 2 | How does Mamba achieve hardware efficiency on ... | Mamba uses a parallel scan implementation with... | Mamba achieves hardware efficiency on modern G... | 5 |
| 3 | Which earlier model family does Mamba descend ... | Mamba builds on the structured state-space seq... | State Space Models (SSMs) | 5 |
| 4 | Name two domains beyond text where SSM-style b... | Audio modeling and genomics have both seen suc... | Based on the passages, two domains beyond text... | 5 |
Closing Thoughts¶
Three failure modes:
- Drafters too similar. If all drafters produce similar drafts, the verifier has no real choice. Tune subset diversity.
- Verifier mis-pick. A bad verifier can pick a worse draft. Always include the original retrieval in one drafter's subset so the worst case is vanilla quality.
- Cost on simple queries. Speculative is wasted on questions where vanilla is already correct. Combine with adaptive routing (Recipe 26) so speculative only fires on hard queries.
Compose with caching (always on), streaming (for the verifier output), and adaptive routing (to gate when speculative fires).