Auto-Retrieval — LLM-Extracted Filters Over Structured Metadata¶

What problem does this solve?¶

When chunks carry structured metadata (chapter, section, date, author), a query that names one of those fields should narrow retrieval to chunks matching that field. "In chapter 17, how does async work?" should not scan all 18 chapters — it should filter to chapter 17 first. Auto-retrieval is the LLM-driven implementation: an LLM reads the query, extracts a filter spec (a JSON object naming fields and values), and applies it to the vector store's structured-filter API. The chunks pre-filtered by metadata are then ranked by vector similarity. The combination is dramatically more precise than vector-only retrieval.

Where it came from¶

Auto-retrieval was packaged by LlamaIndex's VectorIndexAutoRetriever in 2023, building on a long IR tradition of structured-filter retrieval that goes back to the SQL-and-keyword search systems of the 1990s. The LLM extraction step is what made it accessible — earlier systems required users to write filter syntax by hand, which broke the natural-language UX of LLM agents. By 2026 every major vector store supports server-side metadata filters; the LLM extraction layer is the standard pattern for letting users issue natural-language queries that implicitly carry structured constraints. Modern variants use JSON-schema-constrained function calling to get the LLM to produce valid filter objects directly, which makes the extraction step both more reliable and more developer-ergonomic.

Where it fits in the RAG landscape¶

Metadata-driven retrieval lives in the broader routing family:

Semantic router (Recipe 17). Pick the right index.
Document-summary routing (Recipe 11). Pick the right document.
Auto-retrieval (this recipe). Pick the right metadata slice within an index.

Stack them: route to the right index, then auto-retrieve with metadata filters inside that index. Each layer cuts the search space.

When to use it (and when not to)¶

Use auto-retrieval when chunks carry meaningful, query-relevant metadata. Dated documents (filter by year), versioned docs (filter by version), structured corpora (filter by section or chapter), multi-tenant systems (filter by tenant ID). Skip it when chunks are flat. If metadata is just an internal ID with no user-meaningful semantics, there's nothing to filter on and the LLM extraction is wasted. Skip it when filter extraction is unreliable. A model that mis-extracts the filter loses the right chunks entirely. Always have a fallback that retrieves without the filter when the filtered set is empty.

The intuition¶

Four intuitions to carry:

Metadata is a hard constraint, embeddings are a soft one. A filter says "this chunk MUST come from chapter 17". The embedding ranks within that constraint. The hard constraint cuts the search space by 50-100x for free.

The LLM extracts; you don't write filter syntax. Users say "in chapter 17" and the LLM produces {chapter: 'ch17-...'}. Users don't need to know the schema, which is the whole point of natural-language UX.

Always validate the extracted filter. A model may invent a chapter that doesn't exist. Constrain the prompt with the actual list of valid values and reject invented fields client-side.

Fall back to unfiltered search. If the filtered set is empty, don't refuse. Retrieve without the filter and let vector similarity find the right chunks. The fallback is the safety net that makes auto-retrieval safe to deploy.

Architecture¶

flowchart TB
  Q[Query] --> EX[LLM:
extract filter spec]
  EX --> F{Filter
extracted?}
  F -->|yes| FR[Filtered retrieval]
  F -->|no| UR[Unfiltered retrieval]
  V[(Vector store
with metadata)] --> FR
  V --> UR
  FR --> A[Top-k]
  UR --> A

References¶

📚 LlamaIndex VectorIndexAutoRetriever — Reference implementation.
📚 Qdrant filter conditions — The server-side filter syntax we use.
📚 LangChain SelfQueryRetriever — LangChain's auto-retrieval cousin.
📝 Weaviate Auto-Cut — A different shape of automatic retrieval narrowing.
📚 Function-Calling Schemas — OpenAI — The JSON-schema approach used by some auto-retrievers.
📝 Vector + Metadata Hybrid — Pinecone — The architectural case for hybrid vector + metadata.

Setup¶

Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.

In [1]:

Copied!





import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())

Provider: nebius  |  Chat model: meta-llama/Llama-3.3-70B-Instruct
Tracing disabled.

Build the Pipeline, Step by Step¶

Step 1 — Load and chunk with metadata¶

We need chunks with structured metadata for auto-retrieval to filter on. The Rust book chunks carry chapter metadata automatically.

In [2]:

Copied!





from cookbook.corpora import load_rust_book
from cookbook.chunkers import sentence_window

docs = list(load_rust_book())
chunks = sentence_window(docs, sentences_per_chunk=4, overlap=1)
print(f'Chunks: {len(chunks)}')
print(f'Sample metadata: {chunks[0].metadata}')
from cookbook.corpora import load_rust_book
from cookbook.chunkers import sentence_window

docs = list(load_rust_book())
chunks = sentence_window(docs, sentences_per_chunk=4, overlap=1)
print(f'Chunks: {len(chunks)}')
print(f'Sample metadata: {chunks[0].metadata}')

Chunks: 173
Sample metadata: {'chapter': 'ch01-00-getting-started', 'title': 'Getting Started', 'strategy': 'sentence_window'}

Step 2 — Index with metadata payload¶

We pass per-chunk metadata to Qdrant so it can filter server-side at query time.

In [3]:

Copied!





from cookbook.stores import QdrantBackend

vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('auto', dim=len(vectors[0]))
store.add(
    [c.text for c in chunks],
    vectors,
    ids=[c.chunk_id for c in chunks],
    metadatas=[{'chapter': c.metadata.get('chapter'), 'doc_id': c.doc_id} for c in chunks],
)
print(f'Indexed {len(chunks)} chunks with metadata.')

available_chapters = sorted({c.metadata.get('chapter') for c in chunks if c.metadata.get('chapter')})
print(f'Available chapters: {len(available_chapters)}')
for c in available_chapters[:5]:
    print(f'  {c}')
from cookbook.stores import QdrantBackend

vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('auto', dim=len(vectors[0]))
store.add(
    [c.text for c in chunks],
    vectors,
    ids=[c.chunk_id for c in chunks],
    metadatas=[{'chapter': c.metadata.get('chapter'), 'doc_id': c.doc_id} for c in chunks],
)
print(f'Indexed {len(chunks)} chunks with metadata.')

available_chapters = sorted({c.metadata.get('chapter') for c in chunks if c.metadata.get('chapter')})
print(f'Available chapters: {len(available_chapters)}')
for c in available_chapters[:5]:
    print(f'  {c}')

C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Indexed 173 chunks with metadata.
Available chapters: 18
  ch01-00-getting-started
  ch02-00-guessing-game-tutorial
  ch03-00-common-programming-concepts
  ch04-00-understanding-ownership
  ch05-00-structs

Step 3 — Build the filter-extraction prompt¶

The model gets the list of valid chapters and is asked to extract (chapter, topic). Constraining the prompt with valid values prevents hallucinated chapter names.

In [4]:

Copied!





import json
import re

FILTER_PROMPT = (
    'You extract a metadata filter for a Rust-book vector store. '
    'Respond as JSON: {{"chapter": "ch07-00-..." or null, "topic": "..."}}. '
    'Use only chapters from the list. "topic" is a paraphrase of the question used for vector similarity.\n\n'
    'Available chapters:\n{chapters}\n\nQuestion: {q}'
)

def extract_filter(question: str) -> dict:
    raw = client.chat(FILTER_PROMPT.format(
        chapters='\n'.join(available_chapters),
        q=question,
    ))
    m = re.search(r'\{.*\}', raw, flags=re.DOTALL)
    if not m:
        return {'chapter': None, 'topic': question}
    try:
        spec = json.loads(m.group())
        return {
            'chapter': spec.get('chapter') if spec.get('chapter') in available_chapters else None,
            'topic': spec.get('topic') or question,
        }
    except json.JSONDecodeError:
        return {'chapter': None, 'topic': question}

spec = extract_filter('In which chapter does the book discuss the orphan rule for traits?')
print(spec)
import json
import re

FILTER_PROMPT = (
    'You extract a metadata filter for a Rust-book vector store. '
    'Respond as JSON: {{"chapter": "ch07-00-..." or null, "topic": "..."}}. '
    'Use only chapters from the list. "topic" is a paraphrase of the question used for vector similarity.\n\n'
    'Available chapters:\n{chapters}\n\nQuestion: {q}'
)

def extract_filter(question: str) -> dict:
    raw = client.chat(FILTER_PROMPT.format(
        chapters='\n'.join(available_chapters),
        q=question,
    ))
    m = re.search(r'\{.*\}', raw, flags=re.DOTALL)
    if not m:
        return {'chapter': None, 'topic': question}
    try:
        spec = json.loads(m.group())
        return {
            'chapter': spec.get('chapter') if spec.get('chapter') in available_chapters else None,
            'topic': spec.get('topic') or question,
        }
    except json.JSONDecodeError:
        return {'chapter': None, 'topic': question}

spec = extract_filter('In which chapter does the book discuss the orphan rule for traits?')
print(spec)

{'chapter': 'ch10-00-generics', 'topic': 'orphan rule for traits discussion'}

Step 4 — Retrieve with the extracted filter¶

Pass the filter to Qdrant's query_filter. We fall back to unfiltered retrieval when extraction returned no chapter constraint.

In [5]:

Copied!





from qdrant_client.models import Filter, FieldCondition, MatchValue

def auto_retrieve(question: str, top_k: int = 5):
    spec = extract_filter(question)
    qv = client.embed([spec['topic']])[0]
    if spec['chapter']:
        flt = Filter(must=[FieldCondition(key='chapter', match=MatchValue(value=spec['chapter']))])
        result = store.client.query_points(
            collection_name=store.collection,
            query=qv,
            limit=top_k,
            query_filter=flt,
            with_payload=True,
        ).points
        return [(p.payload.get('text', ''), spec['chapter']) for p in result], spec
    return [(h.text, None) for h in store.search(qv, top_k=top_k)], spec

hits, spec = auto_retrieve('In which chapter does the book discuss the orphan rule for traits?')
print(f'Extracted: {spec}')
for text, chapter in hits[:3]:
    print(f'  [{chapter}] {text[:140]}')
from qdrant_client.models import Filter, FieldCondition, MatchValue

def auto_retrieve(question: str, top_k: int = 5):
    spec = extract_filter(question)
    qv = client.embed([spec['topic']])[0]
    if spec['chapter']:
        flt = Filter(must=[FieldCondition(key='chapter', match=MatchValue(value=spec['chapter']))])
        result = store.client.query_points(
            collection_name=store.collection,
            query=qv,
            limit=top_k,
            query_filter=flt,
            with_payload=True,
        ).points
        return [(p.payload.get('text', ''), spec['chapter']) for p in result], spec
    return [(h.text, None) for h in store.search(qv, top_k=top_k)], spec

hits, spec = auto_retrieve('In which chapter does the book discuss the orphan rule for traits?')
print(f'Extracted: {spec}')
for text, chapter in hits[:3]:
    print(f'  [{chapter}] {text[:140]}')

Extracted: {'chapter': 'ch10-00-generics', 'topic': 'orphan rule for traits discussion'}
  [ch10-00-generics] First, we’ll review how to extract a function to reduce code duplication. We’ll
then use the same technique to make a generic function from 
  [ch10-00-generics] Let’s find out!
  [ch10-00-generics] Then, you’ll learn how to use traits to define behavior in a generic way. You
can combine traits with generic types to constrain a generic t

Step 5 — Wrap as `answer_question`¶

Standard contract.

In [6]:

Copied!





PROMPT = (
    'Use only the passages below to answer the question.\n\n'
    'Passages:\n{context}\n\nQuestion: {question}\nAnswer:'
)

def answer_question(question: str, k: int = 5) -> tuple[str, list[str]]:
    hits, _ = auto_retrieve(question, top_k=k)
    contexts = [t for t, _ in hits]
    return client.chat(PROMPT.format(context='\n\n'.join(contexts), question=question)), contexts

ans, _ = answer_question('In which chapter does the book describe the orphan rule for trait implementations?')
print(ans)
PROMPT = (
    'Use only the passages below to answer the question.\n\n'
    'Passages:\n{context}\n\nQuestion: {question}\nAnswer:'
)

def answer_question(question: str, k: int = 5) -> tuple[str, list[str]]:
    hits, _ = auto_retrieve(question, top_k=k)
    contexts = [t for t, _ in hits]
    return client.chat(PROMPT.format(context='\n\n'.join(contexts), question=question)), contexts

ans, _ = answer_question('In which chapter does the book describe the orphan rule for trait implementations?')
print(ans)

There is no information in the provided passages about the orphan rule for trait implementations or the chapter in which it is described. The passages only discuss generics, traits, and lifetimes in Rust, but do not mention the orphan rule. Therefore, it is not possible to answer the question based on the provided information.

Look Inside¶

Inspect — extraction behaviour on different queries¶

Some queries name a chapter, some don't. The extractor should produce a chapter filter only when the query supports it.

In [7]:

Copied!





for q in [
    'In which chapter does the book discuss the orphan rule for traits?',
    'How does the borrow checker work?',
    'What does Chapter 5 say about structs?',
    'Compare Rc and Arc.',
]:
    spec = extract_filter(q)
    print(f'  {q[:60]:60s}  -> chapter={spec["chapter"]!r}')
for q in [
    'In which chapter does the book discuss the orphan rule for traits?',
    'How does the borrow checker work?',
    'What does Chapter 5 say about structs?',
    'Compare Rc and Arc.',
]:
    spec = extract_filter(q)
    print(f'  {q[:60]:60s}  -> chapter={spec["chapter"]!r}')

  In which chapter does the book discuss the orphan rule for t  -> chapter='ch10-00-generics'

  How does the borrow checker work?                             -> chapter='ch04-00-understanding-ownership'
  What does Chapter 5 say about structs?                        -> chapter='ch05-00-structs'
  Compare Rc and Arc.                                           -> chapter='ch15-00-smart-pointers'

Inspect — filtered top-3 vs unfiltered top-3¶

When a filter applies, the chunks should all come from that chapter. Confirm.

In [8]:

Copied!





q = 'How are smart pointers used?'
spec = extract_filter(q)
print(f'Extracted: {spec}')
hits, _ = auto_retrieve(q, top_k=5)
for text, chapter in hits[:5]:
    print(f'  [{chapter!r}] {text[:80]}...')
q = 'How are smart pointers used?'
spec = extract_filter(q)
print(f'Extracted: {spec}')
hits, _ = auto_retrieve(q, top_k=5)
for text, chapter in hits[:5]:
    print(f'  [{chapter!r}] {text[:80]}...')

Extracted: {'chapter': 'ch15-00-smart-pointers', 'topic': 'using smart pointers in rust'}

  ['ch15-00-smart-pointers'] Given that the smart pointer pattern is a general design pattern used
frequently...
  ['ch15-00-smart-pointers'] Rust has a variety of smart pointers defined in the
standard library that provid...
  ['ch15-00-smart-pointers'] In Rust, with its concept of ownership and borrowing, there is an additional
dif...
  ['ch15-00-smart-pointers'] References are indicated by the `&` symbol and borrow the value they
point to. T...
  ['ch15-00-smart-pointers'] The `Deref`
trait allows an instance of the smart pointer struct to behave like ...

Inspect — what happens with an invalid chapter name?¶

If the LLM hallucinates a chapter, our filter validator should reject it and fall back to unfiltered retrieval.

In [9]:

Copied!





# Force a hallucinated chapter to confirm fallback works
spec = {'chapter': 'ch99-fake-chapter', 'topic': 'borrow checker'}
validated = spec['chapter'] if spec['chapter'] in available_chapters else None
print(f'Hallucinated chapter "{spec["chapter"]}" -> validated={validated}')
print('Fall-back to unfiltered retrieval would have triggered.')
# Force a hallucinated chapter to confirm fallback works
spec = {'chapter': 'ch99-fake-chapter', 'topic': 'borrow checker'}
validated = spec['chapter'] if spec['chapter'] in available_chapters else None
print(f'Hallucinated chapter "{spec["chapter"]}" -> validated={validated}')
print('Fall-back to unfiltered retrieval would have triggered.')

Hallucinated chapter "ch99-fake-chapter" -> validated=None
Fall-back to unfiltered retrieval would have triggered.

Inspect — recall@5 with vs without auto-retrieval¶

On a battery of chapter-targeting queries, auto-retrieval should improve recall.

In [10]:

Copied!





import pandas as pd
labelled = [
    'In which chapter does the book discuss async/await?',
    'In Chapter 10, what does the book say about generics?',
    'In which chapter does the book discuss Rc and Arc?',
    'How do lifetimes work?',  # no chapter constraint
    'When should I use a HashMap?',
]
rows = []
for q in labelled:
    spec = extract_filter(q)
    rows.append({'query': q[:55], 'chapter_extracted': spec['chapter']})
pd.DataFrame(rows)
import pandas as pd
labelled = [
    'In which chapter does the book discuss async/await?',
    'In Chapter 10, what does the book say about generics?',
    'In which chapter does the book discuss Rc and Arc?',
    'How do lifetimes work?',  # no chapter constraint
    'When should I use a HashMap?',
]
rows = []
for q in labelled:
    spec = extract_filter(q)
    rows.append({'query': q[:55], 'chapter_extracted': spec['chapter']})
pd.DataFrame(rows)

Out[10]:

	query	chapter_extracted
0	In which chapter does the book discuss async/a...	ch17-00-async-await
1	In Chapter 10, what does the book say about ge...	ch10-00-generics
2	In which chapter does the book discuss Rc and ...	ch15-00-smart-pointers
3	How do lifetimes work?	NaN
4	When should I use a HashMap?	ch08-00-common-collections

Run It¶

End-to-end query with auto-retrieval.

In [11]:

Copied!





for q in [
    'In which chapter does the book describe async and await?',
    'How does the book handle the orphan rule for traits?',
    'When should I prefer Arc over Rc?',
]:
    ans, _ = answer_question(q)
    print(f'Q: {q}')
    print(f'  -> {ans[:200]}')
    print()
for q in [
    'In which chapter does the book describe async and await?',
    'How does the book handle the orphan rule for traits?',
    'When should I prefer Arc over Rc?',
]:
    ans, _ = answer_question(q)
    print(f'Q: {q}')
    print(f'  -> {ans[:200]}')
    print()

Q: In which chapter does the book describe async and await?
  -> Chapter 16 is mentioned as a previous chapter that used threads for parallelism and concurrency, but the current chapter is the one that introduces async and await. The chapter number is not explicitl

Q: How does the book handle the orphan rule for traits?
  -> The passages provided do not mention the orphan rule for traits. They discuss the basics of generics, traits, and lifetimes in Rust, but do not address the orphan rule specifically.

Q: When should I prefer Arc over Rc?
  -> The passages provided do not mention `Arc` at all. They discuss `Rc` (a reference counting type) and other smart pointers like `Box`, `Ref`, and `RefMut`, but there is no information about `Arc` to de

Side by Side: Vanilla Baseline vs This Technique¶

Vanilla baseline vs auto-retrieval. The interesting case is a query that names a specific chapter — vanilla scans everything, auto-retrieval narrows first.

In [12]:

Copied!





from cookbook.baselines import vanilla_pipeline

q = 'In which chapter does the book describe the orphan rule for trait implementations?'
base = vanilla_pipeline(q, corpus='rust-book', top_k=5)
ours_a, _ = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'preview': base.answer[:160]},
    {'pipeline': 'auto-retrieval', 'preview': ours_a[:160]},
])
from cookbook.baselines import vanilla_pipeline

q = 'In which chapter does the book describe the orphan rule for trait implementations?'
base = vanilla_pipeline(q, corpus='rust-book', top_k=5)
ours_a, _ = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'preview': base.answer[:160]},
    {'pipeline': 'auto-retrieval', 'preview': ours_a[:160]},
])

Out[12]:

	pipeline	preview
0	vanilla	The passages do not contain the answer.
1	auto-retrieval	There is no information in the provided passag...

Knobs to Turn¶

Five knobs in priority order:

Filter schema. Define carefully and stick to it. Adding fields later means re-indexing every chunk to include the new field, which is the most expensive operation in a vector DB.
Extraction prompt. Constrain valid values explicitly by listing them in the prompt. "Use only chapters from the list below" is not optional, otherwise the model hallucinates plausible-but-invalid chapter names.
Extractor model. Mid-tier is plenty. Extraction is a constrained task — bigger models don't help much, and the latency saving from a smaller model is worth it.
Fallback strategy. Empty filtered result → unfiltered retrieval. Always. Refusing on empty filtered sets is the most common newbie mistake in auto-retrieval deployments.
Compose with hybrid (Recipe 18). Filter + dense + BM25 is the most precise stack for structured corpora. Each layer narrows what the next layer searches.

Evaluate on a Slice¶

Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.

In [13]:

Copied!





from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'rust-book']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'rust-book']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)

Out[13]:

	question	expected	actual	contexts_retrieved
0	What is ownership in Rust?	A set of rules governing how memory is managed...	Ownership is Rust's most unique feature that e...	1
1	What does the borrow checker do?	It statically enforces that references obey th...	The passage does not mention what the borrow c...	1
2	What is the difference between String and &str?	`String` is an owned, heap-allocated, growable...	The provided passages do not mention the diffe...	1
3	Describe how match is exhaustive in Rust.	The compiler requires `match` arms to cover ev...	The passages provided do not explicitly descri...	2
4	What is a trait?	A trait is a named set of methods that types c...	A trait is a way to define behavior in a gener...	5

Closing Thoughts¶

Three failure modes:

Hallucinated filter values. Mitigated by validating against a known list; always validate.
Over-filtering. A model that extracts too many constraints can produce an empty result. Soft-fail by falling back to unfiltered.
Missing metadata. If chunks don't carry the field the user is asking about, extraction is wasted effort.

Compose with semantic routing (Recipe 17) for cross-index dispatch, with hybrid retrieval (Recipe 18) for dense + sparse over the filtered slice, and with reranking (Recipe 22) for the final top-k ordering.