Self-Reflective Retrieval — Letting the Model Decide¶

What problem does this solve?¶

Vanilla RAG retrieves on every query, regardless of whether retrieval helps. For trivia the model already knows, retrieval injects noise. For questions outside the corpus, retrieval drags the model toward confidently-wrong neighbours. For multi-hop questions, a single retrieval pass is structurally insufficient. The fix is to let the model decide — should I retrieve, and is what I got any good? Self-RAG (Asai et al., ICLR 2024) trained a model that emits reflection tokens during generation: a [Retrieve] decision before retrieving, a [IsRel] decision after each retrieved passage, a [IsSup] decision over whether the answer is supported. The published recipes use the trained model; this notebook reproduces the behaviour with prompt-engineered reflection so it works on any base model — Llama, Qwen, GPT-4o, Claude — without specialised weights.

Where it came from¶

Self-RAG was published at ICLR 2024 by Akari Asai and colleagues at the University of Washington. The paper showed that explicit reflection tokens, trained via instruction-tuning and a custom critic model, beat both vanilla RAG and chain-of-thought RAG on TriviaQA, PubHealth, and ARC-Challenge. The headline number was a 4–7 point gain on PubHealth grounding, but the more important contribution was the framing: retrieval is a decision, not a default. By mid-2025 the prompt-engineered variant — ask the base model the same yes/no questions — was the dominant production pattern, because it works on any model and the quality gap to the fine-tuned variant is small. We use the prompt variant here so the recipe runs on any provider.

Where it fits in the RAG landscape¶

Three cousins in this category, each making a different decision explicit:

Self-RAG (this recipe) — should I retrieve at all, and are the passages useful? Reflection happens per query and per passage.
CRAG (Recipe 25) — given that I retrieved, are the passages good enough? If not, fall back to web search. Reflection happens after retrieval, before generation.
Adaptive-RAG (Recipe 26) — classify the question into {no retrieval, single-shot, multi-hop} and dispatch. Reflection happens before retrieval, on the query alone.

Real production systems often stack all three. Adaptive-RAG routes; Self-RAG decides per-passage usefulness; CRAG catches the case where every passage is unhelpful. The cost is three extra LLM calls per query; the payoff is sharply lower hallucination and much better behaviour on out-of-corpus questions.

When to use it (and when not to)¶

Use Self-RAG when your corpus covers some but not all of what users will ask. Customer-support knowledge bases, internal wikis, technical documentation — anywhere the user might reasonably ask something the corpus does not address. The reflection layer turns those queries into honest "I don't have evidence for that" answers instead of confident hallucinations. Skip it when your corpus is exhaustive and on-topic by construction. A FAQ retrieval system over a closed Q&A set rarely needs reflection — the answer is always there. The extra LLM calls just add latency. Skip it also when latency matters more than quality. Each reflection adds 200–500 ms; on a fast model that is fine, on a slower model it adds up. Recipe 27 (Speculative RAG) is the right answer when you need both reflection and low latency.

The intuition¶

Three intuitions that explain why this works:

Models are honest when explicitly asked. Ask a model "Is this passage useful for answering X?" with a yes/no constraint and it tells you. Ask it to answer X with that passage in context and it will use the passage even when it should not. The reflection prompt is a different lens than the generation prompt.

Per-passage filtering scales gracefully. k=10 retrieval with passage-level filtering routinely beats k=5 retrieval without. You can be generous with retrieval recall as long as the reflection layer is cheap enough to scale linearly.

Refusals are a feature, not a bug. A system that says "I cannot answer this confidently from the corpus" is more trustworthy than one that hallucinates. Self-RAG buys you that refusal capability without changing the base model.

Architecture¶

flowchart TB
  Q[User question] --> DEC{Retrieve?
LLM yes/no}
  DEC -->|no| ANS1[Answer
from world knowledge]
  DEC -->|yes| R[Retrieve top-k]
  R --> F{For each passage:
useful?}
  F -->|drop| F
  F -->|keep| K[Useful passages]
  K --> GEN[Answer using
kept passages]
  GEN --> SUP{Is the answer
supported?}
  SUP -->|yes| OUT[Return answer]
  SUP -->|no| RETRY[Retry or refuse]
  ANS1 --> OUT

References¶

📄 Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — The original ICLR 2024 paper from Asai et al.
💻 Self-RAG official repository — Trained reflection-token weights and training scripts.
📚 LangGraph Self-RAG tutorial — Reference implementation as a stateful graph.
📄 CRAG: Corrective Retrieval Augmented Generation — Related work; covered in Recipe 25.
📄 Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity — The third cousin; covered in Recipe 26.
📝 The Self-RAG paper, one-page summary — Useful illustrated walk-through of the reflection tokens.

Setup¶

Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.

In [1]:

Copied!





import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())

Provider: nebius  |  Chat model: meta-llama/Llama-3.3-70B-Instruct
Tracing disabled.

Build the Pipeline, Step by Step¶

Step 1 — Set up the corpus and an index¶

We use the Rust book chapters. The corpus deliberately covers some questions (Rust syntax, ownership, async) but not others (assembly, Python, history of programming languages). That mix lets us exercise the reflection logic on both kinds of queries.

In [2]:

Copied!





from cookbook.corpora import load_rust_book
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend

docs = list(load_rust_book())
chunks = sentence_window(docs, sentences_per_chunk=4, overlap=1)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('selfrag', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks from {len(docs)} chapters.')
from cookbook.corpora import load_rust_book
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend

docs = list(load_rust_book())
chunks = sentence_window(docs, sentences_per_chunk=4, overlap=1)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('selfrag', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks from {len(docs)} chapters.')

C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Indexed 173 chunks from 18 chapters.

Step 2 — The `[Retrieve?]` decision¶

Before any retrieval, ask the model whether this corpus is relevant to the question. We constrain the response to exactly two tokens so parsing is trivial. This is the cheapest possible reflection step and it pays for itself on out-of-corpus questions.

In [3]:

Copied!





RETRIEVE_DECISION = (
    'You have access to a knowledge base about the Rust programming language. '
    'For the following question, would consulting that knowledge base materially help? '
    'Reply with exactly one word: RETRIEVE or NO_RETRIEVE.\n\n'
    'Question: {q}\n\nAnswer:'
)

def needs_retrieval(question: str) -> bool:
    out = client.chat(RETRIEVE_DECISION.format(q=question)).strip().upper()
    return 'NO_RETRIEVE' not in out.split()[:2]

probes = [
    'What is the borrow checker in Rust?',
    'Who wrote The Great Gatsby?',
    'How do I send a value across threads safely in Rust?',
    'What is the chemical formula for water?',
]
for p in probes:
    print(f'  {needs_retrieval(p)!s:5s}  -> {p}')
RETRIEVE_DECISION = (
    'You have access to a knowledge base about the Rust programming language. '
    'For the following question, would consulting that knowledge base materially help? '
    'Reply with exactly one word: RETRIEVE or NO_RETRIEVE.\n\n'
    'Question: {q}\n\nAnswer:'
)

def needs_retrieval(question: str) -> bool:
    out = client.chat(RETRIEVE_DECISION.format(q=question)).strip().upper()
    return 'NO_RETRIEVE' not in out.split()[:2]

probes = [
    'What is the borrow checker in Rust?',
    'Who wrote The Great Gatsby?',
    'How do I send a value across threads safely in Rust?',
    'What is the chemical formula for water?',
]
for p in probes:
    print(f'  {needs_retrieval(p)!s:5s}  -> {p}')

  True   -> What is the borrow checker in Rust?
  False  -> Who wrote The Great Gatsby?
  True   -> How do I send a value across threads safely in Rust?
  False  -> What is the chemical formula for water?

The model should say True for Rust questions and False for the others. If it says True for everything, the prompt is too lenient — tighten the wording. If it says False for Rust questions, the prompt is too strict.

Step 3 — Retrieve with k=8 (generous recall)¶

With reflection filtering downstream, we can afford generous initial recall. We pull eight candidates and let the next step drop unhelpful ones. Without reflection we would stick to k=5 to keep the prompt tight.

In [4]:

Copied!





def retrieve_candidates(question: str, k: int = 8):
    qv = client.embed([question])[0]
    return store.search(qv, top_k=k)

candidates = retrieve_candidates('How does the borrow checker enforce exclusive mutable references?')
for i, h in enumerate(candidates, 1):
    print(f'  {i}. score={h.score:.3f}  {h.text[:140]}')
def retrieve_candidates(question: str, k: int = 8):
    qv = client.embed([question])[0]
    return store.search(qv, top_k=k)

candidates = retrieve_candidates('How does the borrow checker enforce exclusive mutable references?')
for i, h in enumerate(candidates, 1):
    print(f'  {i}. score={h.score:.3f}  {h.text[:140]}')

  1. score=0.647  Given that the smart pointer pattern is a general design pattern used
frequently in Rust, this chapter won’t cover every existing smart poin
  2. score=0.618  # Understanding Ownership

Ownership is Rust’s most unique feature and has deep implications for the rest
of the language. It enables Rust t
  3. score=0.563  References are a complex feature,
and one of Rust’s major advantages is how safe and easy it is to use
references. You don’t need to know a 
  4. score=0.552  The full job of `read_line` is to take
whatever the user types into standard input and append that into a string
(without overwriting its co
  5. score=0.550  Initially, the Rust team thought that ensuring memory safety and preventing
concurrency problems were two separate challenges to be solved w
  6. score=0.540  Hence, you need to write `&mut guess` rather than
`&guess` to make it mutable. (Chapter 4 will explain references more
thoroughly.)

<!-- Ol
  7. score=0.534  This function’s signature accepts an integer as a parameter and returns an
integer as a result. When we implement and compile that function,
  8. score=0.528  In Rust, variables are immutable by default, meaning once we give the variable
a value, the value won’t change. We’ll be discussing this con

Step 4 — The `[IsRel]` per-passage filter¶

For each candidate passage, ask the model whether it materially helps answer the question. Drop passages that get NOT_USEFUL. This is where the bulk of the reflection budget is spent — k LLM calls per query — but each call is short and the impact on quality is large.

In [5]:

Copied!





USEFUL_DECISION = (
    'Question: {q}\n\n'
    'Passage: {p}\n\n'
    'Does this passage materially help answer the question? '
    'Reply with exactly one word: USEFUL or NOT_USEFUL.'
)

def filter_useful(question: str, candidates) -> list:
    kept = []
    for h in candidates:
        verdict = client.chat(USEFUL_DECISION.format(q=question, p=h.text[:800])).strip().upper()
        if 'USEFUL' in verdict and 'NOT_USEFUL' not in verdict:
            kept.append(h)
    return kept

question = 'How does the borrow checker enforce exclusive mutable references?'
kept = filter_useful(question, candidates)
print(f'Kept {len(kept)} of {len(candidates)} candidates.')
for h in kept:
    print(f'  - {h.text[:140]}')
USEFUL_DECISION = (
    'Question: {q}\n\n'
    'Passage: {p}\n\n'
    'Does this passage materially help answer the question? '
    'Reply with exactly one word: USEFUL or NOT_USEFUL.'
)

def filter_useful(question: str, candidates) -> list:
    kept = []
    for h in candidates:
        verdict = client.chat(USEFUL_DECISION.format(q=question, p=h.text[:800])).strip().upper()
        if 'USEFUL' in verdict and 'NOT_USEFUL' not in verdict:
            kept.append(h)
    return kept

question = 'How does the borrow checker enforce exclusive mutable references?'
kept = filter_useful(question, candidates)
print(f'Kept {len(kept)} of {len(candidates)} candidates.')
for h in kept:
    print(f'  - {h.text[:140]}')

Kept 0 of 8 candidates.

Usually 3–5 of the 8 candidates survive. The dropped passages tend to be marginally on-topic — they mention the right keywords but discuss a different aspect of the topic. Compare the dropped passages to the kept ones manually to build intuition for what your model considers "useful".

Step 5 — Generate from the surviving passages¶

Standard stuffed-context generation, but now only over passages that survived the filter. If zero passages survived, refuse honestly rather than answer from nothing.

In [6]:

Copied!





GENERATE_PROMPT = (
    'Use ONLY the passages below to answer the question. '
    'If they do not contain the answer, say so plainly.\n\n'
    'Passages:\n{context}\n\nQuestion: {question}\nAnswer:'
)

def generate_answer(question: str, passages: list) -> str:
    if not passages:
        return 'No useful evidence retrieved. Cannot answer from this corpus.'
    context = '\n\n'.join(p.text for p in passages)
    return client.chat(GENERATE_PROMPT.format(context=context, question=question))

answer = generate_answer(question, kept)
print(answer)
GENERATE_PROMPT = (
    'Use ONLY the passages below to answer the question. '
    'If they do not contain the answer, say so plainly.\n\n'
    'Passages:\n{context}\n\nQuestion: {question}\nAnswer:'
)

def generate_answer(question: str, passages: list) -> str:
    if not passages:
        return 'No useful evidence retrieved. Cannot answer from this corpus.'
    context = '\n\n'.join(p.text for p in passages)
    return client.chat(GENERATE_PROMPT.format(context=context, question=question))

answer = generate_answer(question, kept)
print(answer)

No useful evidence retrieved. Cannot answer from this corpus.

Step 6 — The `[IsSup]` support check¶

Ask the model whether its own answer is supported by the passages it used. This catches the case where the model wrote a confident-sounding answer that drifted beyond the retrieved evidence. The check is one LLM call; on production systems it gates the response.

In [7]:

Copied!





SUPPORT_CHECK = (
    'Passages:\n{context}\n\n'
    'Answer: {answer}\n\n'
    'Is every factual claim in the answer supported by the passages? '
    'Reply with exactly one word: SUPPORTED or UNSUPPORTED.'
)

def check_support(answer: str, passages: list) -> bool:
    if not passages:
        return False
    context = '\n\n'.join(p.text for p in passages)
    verdict = client.chat(SUPPORT_CHECK.format(context=context, answer=answer)).strip().upper()
    return 'SUPPORTED' in verdict and 'UNSUPPORTED' not in verdict

supported = check_support(answer, kept)
print(f'Answer supported by retrieved passages: {supported}')
SUPPORT_CHECK = (
    'Passages:\n{context}\n\n'
    'Answer: {answer}\n\n'
    'Is every factual claim in the answer supported by the passages? '
    'Reply with exactly one word: SUPPORTED or UNSUPPORTED.'
)

def check_support(answer: str, passages: list) -> bool:
    if not passages:
        return False
    context = '\n\n'.join(p.text for p in passages)
    verdict = client.chat(SUPPORT_CHECK.format(context=context, answer=answer)).strip().upper()
    return 'SUPPORTED' in verdict and 'UNSUPPORTED' not in verdict

supported = check_support(answer, kept)
print(f'Answer supported by retrieved passages: {supported}')

Answer supported by retrieved passages: False

Step 7 — Glue it all together¶

The whole Self-RAG loop in one function. This is what the rest of the cookbook compares against the vanilla baseline.

In [8]:

Copied!





def answer_question(question: str) -> tuple[str, list[str]]:
    if not needs_retrieval(question):
        free = client.chat(f'Answer concisely from your own knowledge: {question}')
        return free, []
    candidates = retrieve_candidates(question, k=8)
    kept = filter_useful(question, candidates)
    answer = generate_answer(question, kept)
    if not check_support(answer, kept):
        answer = 'I do not have sufficient evidence in the corpus to answer that confidently.\n\n' + answer
    return answer, [h.text for h in kept]

ans, ctxs = answer_question('What is interior mutability and how does RefCell expose it?')
print(ans)
print()
print(f'(used {len(ctxs)} passages)')
def answer_question(question: str) -> tuple[str, list[str]]:
    if not needs_retrieval(question):
        free = client.chat(f'Answer concisely from your own knowledge: {question}')
        return free, []
    candidates = retrieve_candidates(question, k=8)
    kept = filter_useful(question, candidates)
    answer = generate_answer(question, kept)
    if not check_support(answer, kept):
        answer = 'I do not have sufficient evidence in the corpus to answer that confidently.\n\n' + answer
    return answer, [h.text for h in kept]

ans, ctxs = answer_question('What is interior mutability and how does RefCell expose it?')
print(ans)
print()
print(f'(used {len(ctxs)} passages)')

The passages do not contain a detailed explanation of what interior mutability is, but they do mention that `RefCell<T>` is a type that enforces the borrowing rules at runtime instead of compile time and is related to the interior mutability pattern, where an immutable type exposes an API for mutating an interior value.

(used 1 passages)

Look Inside¶

Inspect — does the `[Retrieve?]` decision behave well across question types?¶

Run a small battery of clearly-on-topic, clearly-off-topic, and ambiguous questions. The decision should be on for the first group, off for the second, and the ambiguous ones tell you where your prompt sits.

In [9]:

Copied!





battery = [
    ('on-topic',  'What is ownership in Rust?'),
    ('on-topic',  'How does async/await work in Rust?'),
    ('off-topic', 'What year was the French Revolution?'),
    ('off-topic', 'What is the chemical formula of water?'),
    ('ambiguous', 'How is concurrency different from parallelism in general?'),
    ('ambiguous', 'Compare garbage collection to manual memory management.'),
]
for kind, q in battery:
    print(f'  {kind:10s}  retrieve={needs_retrieval(q)!s:5s}  q={q}')
battery = [
    ('on-topic',  'What is ownership in Rust?'),
    ('on-topic',  'How does async/await work in Rust?'),
    ('off-topic', 'What year was the French Revolution?'),
    ('off-topic', 'What is the chemical formula of water?'),
    ('ambiguous', 'How is concurrency different from parallelism in general?'),
    ('ambiguous', 'Compare garbage collection to manual memory management.'),
]
for kind, q in battery:
    print(f'  {kind:10s}  retrieve={needs_retrieval(q)!s:5s}  q={q}')

  on-topic    retrieve=True   q=What is ownership in Rust?
  on-topic    retrieve=True   q=How does async/await work in Rust?
  off-topic   retrieve=False  q=What year was the French Revolution?

  off-topic   retrieve=False  q=What is the chemical formula of water?
  ambiguous   retrieve=True   q=How is concurrency different from parallelism in general?
  ambiguous   retrieve=True   q=Compare garbage collection to manual memory management.

Inspect — what fraction of passages survive the [IsRel] filter?¶

Across several questions, see how aggressive the filter is. A typical Self-RAG run drops 30–60 % of candidates. If your filter keeps everything, it is too lenient and saves you nothing. If it drops everything, it is too strict and the model will refuse perfectly answerable questions.

In [10]:

Copied!





import statistics
rates = []
for q in [
    'How does the borrow checker enforce exclusive references?',
    'What is Rc and when should I use it?',
    'When should I pick Arc over Mutex?',
    'What is the difference between String and &str?',
    'How does Rust handle error propagation with the question mark operator?',
]:
    cs = retrieve_candidates(q, k=8)
    kept = filter_useful(q, cs)
    rates.append(len(kept) / len(cs))
    print(f'  kept {len(kept):2d}/{len(cs)} for: {q[:60]}')
print()
print(f'Mean keep rate: {statistics.mean(rates):.0%}')
import statistics
rates = []
for q in [
    'How does the borrow checker enforce exclusive references?',
    'What is Rc and when should I use it?',
    'When should I pick Arc over Mutex?',
    'What is the difference between String and &str?',
    'How does Rust handle error propagation with the question mark operator?',
]:
    cs = retrieve_candidates(q, k=8)
    kept = filter_useful(q, cs)
    rates.append(len(kept) / len(cs))
    print(f'  kept {len(kept):2d}/{len(cs)} for: {q[:60]}')
print()
print(f'Mean keep rate: {statistics.mean(rates):.0%}')

  kept  0/8 for: How does the borrow checker enforce exclusive references?

  kept  1/8 for: What is Rc and when should I use it?

  kept  0/8 for: When should I pick Arc over Mutex?

  kept  1/8 for: What is the difference between String and &str?
  kept  0/8 for: How does Rust handle error propagation with the question mar

Mean keep rate: 5%

Inspect — what happens on an out-of-corpus question?¶

Ask something the Rust book genuinely does not cover. The whole Self-RAG loop should either skip retrieval or refuse gracefully — not hallucinate.

In [11]:

Copied!





out_of_corpus_q = 'What is the relationship between Hamiltonian mechanics and Lagrangian mechanics?'
ans, used = answer_question(out_of_corpus_q)
print(ans)
print()
print(f'(used {len(used)} passages)')
out_of_corpus_q = 'What is the relationship between Hamiltonian mechanics and Lagrangian mechanics?'
ans, used = answer_question(out_of_corpus_q)
print(ans)
print()
print(f'(used {len(used)} passages)')

Hamiltonian mechanics and Lagrangian mechanics are two equivalent formulations of classical mechanics. They describe the same physical systems, but use different mathematical approaches. Lagrangian mechanics uses the Lagrangian function (kinetic energy minus potential energy) to derive the equations of motion, while Hamiltonian mechanics uses the Hamiltonian function (total energy) to derive the equations of motion. They are related by a Legendre transformation, which allows for the conversion between the two formulations.

(used 0 passages)

Two acceptable behaviours: (1) the [Retrieve?] step returns NO_RETRIEVE and the model answers from world knowledge — no retrieval noise, no refusal; (2) the model retrieves, every passage gets dropped by [IsRel], and the system refuses honestly. The unacceptable behaviour — confident hallucination with cited-but-irrelevant Rust passages — is what Self-RAG prevents.

Inspect — cost breakdown per query¶

A reflection pipeline does more LLM work than vanilla RAG. Quantify it so you know what you are spending. We count the calls in a single end-to-end answer.

In [12]:

Copied!





from cookbook import _cache
before = _cache.stats()['entries']
_ = answer_question('Walk through how lifetimes work in a function that returns a reference to its longest argument.')
after = _cache.stats()['entries']
calls = after - before
print(f'New cache entries this query: {calls}')
print('Rough breakdown:')
print('  1   needs_retrieval decision')
print('  1   query embedding')
print('  k   per-passage IsRel checks  (k=8 -> 8)')
print('  1   final generation')
print('  1   IsSup check')
print(f'  ~{1+1+8+1+1} = 12 calls in the worst case')
from cookbook import _cache
before = _cache.stats()['entries']
_ = answer_question('Walk through how lifetimes work in a function that returns a reference to its longest argument.')
after = _cache.stats()['entries']
calls = after - before
print(f'New cache entries this query: {calls}')
print('Rough breakdown:')
print('  1   needs_retrieval decision')
print('  1   query embedding')
print('  k   per-passage IsRel checks  (k=8 -> 8)')
print('  1   final generation')
print('  1   IsSup check')
print(f'  ~{1+1+8+1+1} = 12 calls in the worst case')

New cache entries this query: 0
Rough breakdown:
  1   needs_retrieval decision
  1   query embedding
  k   per-passage IsRel checks  (k=8 -> 8)
  1   final generation
  1   IsSup check
  ~12 = 12 calls in the worst case

Run It¶

End-to-end on a representative Rust question, so a reader sees the full pipeline output.

In [13]:

Copied!





q = 'When should I prefer Arc over Rc, and what guarantees do I lose if I switch?'
ans, ctxs = answer_question(q)
print('=== Self-RAG answer ===')
print(ans)
print()
print(f'(used {len(ctxs)} passages)')
q = 'When should I prefer Arc over Rc, and what guarantees do I lose if I switch?'
ans, ctxs = answer_question(q)
print('=== Self-RAG answer ===')
print(ans)
print()
print(f'(used {len(ctxs)} passages)')

=== Self-RAG answer ===
I do not have sufficient evidence in the corpus to answer that confidently.

No useful evidence retrieved. Cannot answer from this corpus.

(used 0 passages)

Side by Side: Vanilla Baseline vs This Technique¶

Show vanilla baseline vs Self-RAG on the same question. The baseline tends to retrieve more aggressively and stuff lower-quality passages; Self-RAG filters them out. Sometimes the answers are similar; sometimes the baseline hallucinates where Self-RAG refuses or stays grounded.

In [14]:

Copied!





from cookbook.baselines import vanilla_pipeline

q = 'When should I prefer Arc over Rc, and what guarantees do I lose if I switch?'
base = vanilla_pipeline(q, corpus='rust-book', top_k=5)
ours_ans, ours_ctxs = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'n_contexts': len(base.contexts), 'preview': base.answer[:160]},
    {'pipeline': 'self-rag', 'n_contexts': len(ours_ctxs), 'preview': ours_ans[:160]},
])
from cookbook.baselines import vanilla_pipeline

q = 'When should I prefer Arc over Rc, and what guarantees do I lose if I switch?'
base = vanilla_pipeline(q, corpus='rust-book', top_k=5)
ours_ans, ours_ctxs = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'n_contexts': len(base.contexts), 'preview': base.answer[:160]},
    {'pipeline': 'self-rag', 'n_contexts': len(ours_ctxs), 'preview': ours_ans[:160]},
])

Out[14]:

	pipeline	n_contexts	preview
0	vanilla	5	The passages do not contain the answer to this...
1	self-rag	0	I do not have sufficient evidence in the corpu...

Knobs to Turn¶

Five knobs, ranked by impact:

The [Retrieve?] prompt. Decides which queries skip retrieval entirely. Loosen the prompt to bias toward retrieval (safer for unknown domains) or tighten it to save tokens (better when your corpus is narrow and questions usually fall outside).
Initial k for retrieval. With per-passage filtering downstream, you can crank k up. We use 8; production systems often go to 12–20.
The [IsRel] prompt strictness. Aggressive filters cut faithfulness errors but raise refusal rate. Tune by measuring both on a held-out eval set.
Use a cheaper model for reflection. The reflection calls do not need your best model. Many production systems use a small fast model (Qwen-7B, GPT-4o-mini, Groq Llama-3.3-70b-instant) for reflection and reserve the bigger model for the final generation.
Reflection caching. Reflection answers depend only on (question, passage). With the cookbook's disk cache on, repeated queries are free; in production, cache keys by hash of (question, passage) for the same payoff.

Evaluate on a Slice¶

Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.

In [15]:

Copied!





from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'rust-book']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'rust-book']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)

Out[15]:

	question	expected	actual	contexts_retrieved
0	What is ownership in Rust?	A set of rules governing how memory is managed...	The passages do not contain a direct definitio...	3
1	What does the borrow checker do?	It statically enforces that references obey th...	I do not have sufficient evidence in the corpu...	0
2	What is the difference between String and &str?	`String` is an owned, heap-allocated, growable...	The passages do not contain the answer to the ...	1
3	Describe how match is exhaustive in Rust.	The compiler requires `match` arms to cover ev...	I do not have sufficient evidence in the corpu...	1
4	What is a trait?	A trait is a named set of methods that types c...	I do not have sufficient evidence in the corpu...	0

Closing Thoughts¶

Three places Self-RAG falls short:

Latency. 8 reflection calls + 1 generation + 1 support check means 10 calls per query. Even at 200 ms each, that is 2 seconds. Recipe 27 (Speculative RAG) is the right answer if you need both reflection and speed.
Drift across calls. The reflection model and the generation model do not always agree. If you generate with Llama-3.3 and reflect with a smaller model, you will see cases where the reflection rejects passages the generator could have used well. Match the reflection model to the generation model when this matters.
Subtle off-topic. A passage that mentions the right keywords but discusses a different sub-topic often slips through [IsRel]. The reflection is yes/no; sometimes the right answer is "partially useful, use for context only". Recipe 22 (cross-encoder rerank) is a better tool when the corpus has lots of near-misses.

Self-RAG remains the cheapest way to add basic safety to a RAG system. Stack it with CRAG (Recipe 25) for the case where every passage fails the filter; stack it with Adaptive-RAG (Recipe 26) for cheap query routing.