Semantic Router — Classify the Query, Dispatch the Index¶

What problem does this solve?¶

A production assistant covers many topics. The user asks one question; the system has many indexes. Sending every query to every index is wasteful and noisy — the top hit from the wrong index can outrank the right hit from the right one. Semantic routing solves the problem cheaply. Define routes by example: "Rust ownership", "borrow checker", "lifetimes" → rust_book. "Cooper pairs", "BCS theory" → superconductors. Embed the examples; at query time, find the route with the highest cosine match. Dispatch the query to that route's index. No LLM call needed.

Where it came from¶

The pattern was crystallised by Aurelio Labs' semantic-router library in 2024. The underlying technique — cosine-classifier with example phrases per class — predates RAG by years (it's how early intent-classification systems worked in dialogue research) but the library made it trivially composable with LLM agents. By 2026 every production RAG system that handles mixed query types uses some form of semantic router or LLM-as-classifier upstream. The semantic-router variant tends to win on cost; the LLM-classifier variant wins on flexibility. Most mature systems stack both — semantic router for fast routing on confident queries, LLM classifier as fall-through.

Where it fits in the RAG landscape¶

Three routing options to know:

Semantic router (this recipe). Cosine match against example phrases per route. Free at query time after the index is built. Best when routes are well-separated.
LLM-as-classifier (Adaptive-RAG, Recipe 26). One LLM call per query. Slower, smarter, more nuanced on ambiguous queries.
Metadata filter (Recipe 20). Structured filters on chunk metadata. Requires labelled metadata to exist on chunks. Free at query time but inflexible.

Stack them: semantic router for coarse routing, LLM-as-classifier as fallback when the router score is low. Metadata filters compose underneath both, narrowing the per-route search space further.

When to use it (and when not to)¶

Use a semantic router any time you have multiple knowledge sources. Multi-tenant systems, multi-product knowledge bases, multi-domain agents — anywhere a query genuinely belongs to one of several distinct indexes. The router is the cheapest decision in your stack. Skip it for single-source systems. There is nothing to route to and the routing cost is wasted. Skip it when routes blur. If users routinely ask questions that span multiple routes ("compare X-related to Y-related"), the router will mis-route or refuse consistently. A higher-level layer like sub-question decomposition (Recipe 16) handles cross-route queries better.

The intuition¶

Four intuitions:

Example phrases define the route. A route is just a set of canonical queries. The more diverse the examples, the better the route generalises. Add new examples as you observe misrouting in production.

Cosine is enough. No need for a learned classifier; the embedder already encodes semantic similarity. Cosine over example embeddings is a strong baseline that holds up against most fancier approaches in head-to-head tests.

Always have a fall-through. When the top score is below a threshold, fall back to the LLM classifier or refuse. Hard-routing every query is brittle and costs you on out-of-distribution queries that will inevitably show up.

Threshold tuning is empirical. There is no theoretical threshold value. Sweep on a labelled out-of-distribution slice and find where false-positive rate flattens.

Architecture¶

flowchart TB
  R1[Route 1
example phrases] --> E1[Embed]
  R2[Route 2
example phrases] --> E2[Embed]
  R3[Route 3
example phrases] --> E3[Embed]
  E1 --> S[(Centroid
or examples)]
  E2 --> S
  E3 --> S
  Q[Query] --> QE[Embed]
  QE --> M[Cosine match
vs routes]
  S --> M
  M --> D{Score >=
threshold?}
  D -->|yes| R[Dispatch]
  D -->|no| F[Fallback /
refuse]

References¶

💻 aurelio-labs/semantic-router — The library that popularised the pattern.
📄 Adaptive-RAG (Recipe 26) — LLM-as-classifier alternative.
📚 LlamaIndex RouterQueryEngine — LlamaIndex's router abstraction.
📄 RAGRouter Bench — Recent benchmark comparing routing approaches.
📚 LangChain Multi-Vector + Routing tutorial — LangChain's routing patterns.
📚 MTEB Classification — Embedding benchmark whose classification task correlates with route accuracy.

Setup¶

Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.

In [1]:

Copied!





import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())

Provider: nebius  |  Chat model: meta-llama/Llama-3.3-70B-Instruct
Tracing disabled.

Build the Pipeline, Step by Step¶

Step 1 — Define routes by example¶

Each route is named and given 4-8 example phrases. The phrases should be diverse — different vocabulary, different aspects of the topic. For this notebook we route across all four cookbook corpora.

In [2]:

Copied!





ROUTES = {
    'arxiv-mamba': [
        'state-space models', 'selective scan', 'linear attention alternative',
        'long sequence modeling', 'recurrent neural network backbone',
    ],
    'wikipedia-superconductors': [
        'Meissner effect', 'Cooper pairs', 'critical temperature', 'flux pinning',
        'BCS theory', 'YBCO', 'high-temperature superconductor',
    ],
    'sec-10k-pltr': [
        'annual report risk factors', 'government contract revenue', 'AIP platform',
        'Gotham Foundry Apollo', 'cybersecurity disclosures', 'segment reporting',
    ],
    'rust-book': [
        'ownership and borrowing', 'borrow checker', 'lifetimes', 'trait objects',
        'async await', 'Rc and Arc', 'pattern matching',
    ],
}
print(f'{len(ROUTES)} routes defined.')
ROUTES = {
    'arxiv-mamba': [
        'state-space models', 'selective scan', 'linear attention alternative',
        'long sequence modeling', 'recurrent neural network backbone',
    ],
    'wikipedia-superconductors': [
        'Meissner effect', 'Cooper pairs', 'critical temperature', 'flux pinning',
        'BCS theory', 'YBCO', 'high-temperature superconductor',
    ],
    'sec-10k-pltr': [
        'annual report risk factors', 'government contract revenue', 'AIP platform',
        'Gotham Foundry Apollo', 'cybersecurity disclosures', 'segment reporting',
    ],
    'rust-book': [
        'ownership and borrowing', 'borrow checker', 'lifetimes', 'trait objects',
        'async await', 'Rc and Arc', 'pattern matching',
    ],
}
print(f'{len(ROUTES)} routes defined.')

4 routes defined.

Step 2 — Embed the examples¶

One embedding per phrase. Store as numpy arrays for fast cosine.

In [3]:

Copied!





import numpy as np

route_vectors = {}
for name, examples in ROUTES.items():
    vecs = np.asarray(client.embed(examples), dtype=np.float32)
    vecs /= np.linalg.norm(vecs, axis=1, keepdims=True).clip(min=1e-9)
    route_vectors[name] = vecs
print('Route vectors ready.')
import numpy as np

route_vectors = {}
for name, examples in ROUTES.items():
    vecs = np.asarray(client.embed(examples), dtype=np.float32)
    vecs /= np.linalg.norm(vecs, axis=1, keepdims=True).clip(min=1e-9)
    route_vectors[name] = vecs
print('Route vectors ready.')

Route vectors ready.

Step 3 — Score a query against each route¶

Embed the query, dot-product with every route's example vectors, take the max within each route. The route with the highest max wins.

In [4]:

Copied!





def route(question: str) -> tuple[str, float]:
    q = np.asarray(client.embed([question])[0], dtype=np.float32)
    q /= np.linalg.norm(q) + 1e-9
    best, best_score = '?', -1.0
    for name, vecs in route_vectors.items():
        score = float((vecs @ q).max())
        if score > best_score:
            best, best_score = name, score
    return best, best_score

for q in [
    'What is the asymptotic complexity of attention?',
    'How does a SQUID work?',
    'What is Apollo in Palantir terminology?',
    'When should I prefer Arc over Rc?',
]:
    print(f'  {q[:60]:60s}  -> {route(q)}')
def route(question: str) -> tuple[str, float]:
    q = np.asarray(client.embed([question])[0], dtype=np.float32)
    q /= np.linalg.norm(q) + 1e-9
    best, best_score = '?', -1.0
    for name, vecs in route_vectors.items():
        score = float((vecs @ q).max())
        if score > best_score:
            best, best_score = name, score
    return best, best_score

for q in [
    'What is the asymptotic complexity of attention?',
    'How does a SQUID work?',
    'What is Apollo in Palantir terminology?',
    'When should I prefer Arc over Rc?',
]:
    print(f'  {q[:60]:60s}  -> {route(q)}')

  What is the asymptotic complexity of attention?               -> ('arxiv-mamba', 0.8000392913818359)
  How does a SQUID work?                                        -> ('wikipedia-superconductors', 0.620593249797821)
  What is Apollo in Palantir terminology?                       -> ('sec-10k-pltr', 0.6792171001434326)
  When should I prefer Arc over Rc?                             -> ('rust-book', 0.8274726867675781)

Step 4 — Dispatch + fall-through¶

When the top score is below a threshold (default 0.4), the router can't confidently route. Fall through to a default behaviour — refuse, ask the user, or LLM-classify.

In [5]:

Copied!





ROUTE_THRESHOLD = 0.4

def route_or_none(question: str) -> str | None:
    name, score = route(question)
    return name if score >= ROUTE_THRESHOLD else None

for q in [
    'What is the Meissner effect?',
    'Tell me about cooking pasta.',
    'When should I use async in Rust?',
]:
    dest = route_or_none(q)
    print(f'  {q[:50]:50s}  -> {dest!r}')
ROUTE_THRESHOLD = 0.4

def route_or_none(question: str) -> str | None:
    name, score = route(question)
    return name if score >= ROUTE_THRESHOLD else None

for q in [
    'What is the Meissner effect?',
    'Tell me about cooking pasta.',
    'When should I use async in Rust?',
]:
    dest = route_or_none(q)
    print(f'  {q[:50]:50s}  -> {dest!r}')

  What is the Meissner effect?                        -> 'wikipedia-superconductors'
  Tell me about cooking pasta.                        -> None
  When should I use async in Rust?                    -> 'rust-book'

Step 5 — Wrap as `answer_question`¶

For the standard cookbook contract, we route and then call into the appropriate corpus loader. We use the vanilla pipeline as the per-route RAG implementation.

In [6]:

Copied!





from cookbook.baselines import vanilla_pipeline

def answer_question(question: str) -> tuple[str, list[str]]:
    dest = route_or_none(question)
    if dest is None:
        return ('I could not confidently route your question.', [])
    result = vanilla_pipeline(question, corpus=dest, top_k=5)
    return result.answer, result.contexts

ans, _ = answer_question('What is YBCO?')
print(ans)
from cookbook.baselines import vanilla_pipeline

def answer_question(question: str) -> tuple[str, list[str]]:
    dest = route_or_none(question)
    if dest is None:
        return ('I could not confidently route your question.', [])
    result = vanilla_pipeline(question, corpus=dest, top_k=5)
    return result.answer, result.contexts

ans, _ = answer_question('What is YBCO?')
print(ans)

C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

YBCO is a family of crystalline chemical compounds that display high-temperature superconductivity, specifically yttrium barium copper oxide.

Look Inside¶

Inspect — score distribution per query¶

For one query, see how confidently the router picked. The gap between top and second tells you whether routing is sharp or shaky.

In [7]:

Copied!





q = 'How does the Meissner effect work?'
qv = np.asarray(client.embed([q])[0], dtype=np.float32)
qv /= np.linalg.norm(qv) + 1e-9
scored = {name: float((vecs @ qv).max()) for name, vecs in route_vectors.items()}
for name, s in sorted(scored.items(), key=lambda x: x[1], reverse=True):
    print(f'  {name:35s}  {s:.3f}')
q = 'How does the Meissner effect work?'
qv = np.asarray(client.embed([q])[0], dtype=np.float32)
qv /= np.linalg.norm(qv) + 1e-9
scored = {name: float((vecs @ qv).max()) for name, vecs in route_vectors.items()}
for name, s in sorted(scored.items(), key=lambda x: x[1], reverse=True):
    print(f'  {name:35s}  {s:.3f}')

  wikipedia-superconductors            0.849
  rust-book                            0.417
  arxiv-mamba                          0.406
  sec-10k-pltr                         0.404

Inspect — out-of-distribution queries¶

Queries that don't fit any route should score below threshold. Make sure the fall-through fires.

In [8]:

Copied!





for q in [
    'What is the capital of France?',
    'Tell me a joke.',
    'Translate hello to Mandarin.',
]:
    _, s = route(q)
    print(f'  {q[:40]:40s}  top-score={s:.3f}  routed={"yes" if s >= ROUTE_THRESHOLD else "no"}')
for q in [
    'What is the capital of France?',
    'Tell me a joke.',
    'Translate hello to Mandarin.',
]:
    _, s = route(q)
    print(f'  {q[:40]:40s}  top-score={s:.3f}  routed={"yes" if s >= ROUTE_THRESHOLD else "no"}')

  What is the capital of France?            top-score=0.251  routed=no
  Tell me a joke.                           top-score=0.452  routed=yes
  Translate hello to Mandarin.              top-score=0.386  routed=no

Inspect — confusion matrix¶

Run a labelled battery and see which queries go where. A confusion matrix highlights routes that overlap.

In [9]:

Copied!





labelled = [
    ('arxiv-mamba', 'What is selective scan?'),
    ('arxiv-mamba', 'How does Mamba differ from a transformer?'),
    ('wikipedia-superconductors', 'What is Cooper pairing?'),
    ('wikipedia-superconductors', 'When was superconductivity discovered?'),
    ('sec-10k-pltr', 'What are Palantir government revenue risks?'),
    ('sec-10k-pltr', 'What is AIP?'),
    ('rust-book', 'When should I use Arc over Rc?'),
    ('rust-book', 'How does borrow checking work?'),
]
correct = sum(1 for label, q in labelled if route(q)[0] == label)
print(f'Routing accuracy on labelled battery: {correct}/{len(labelled)}')
labelled = [
    ('arxiv-mamba', 'What is selective scan?'),
    ('arxiv-mamba', 'How does Mamba differ from a transformer?'),
    ('wikipedia-superconductors', 'What is Cooper pairing?'),
    ('wikipedia-superconductors', 'When was superconductivity discovered?'),
    ('sec-10k-pltr', 'What are Palantir government revenue risks?'),
    ('sec-10k-pltr', 'What is AIP?'),
    ('rust-book', 'When should I use Arc over Rc?'),
    ('rust-book', 'How does borrow checking work?'),
]
correct = sum(1 for label, q in labelled if route(q)[0] == label)
print(f'Routing accuracy on labelled battery: {correct}/{len(labelled)}')

Routing accuracy on labelled battery: 8/8

Inspect — what happens if I add more examples to a route?¶

Routes get sharper with more examples. We test by adding 5 more examples to one route and re-scoring.

In [10]:

Copied!





extra = ['superconducting magnet', 'persistent current', 'flux quantum', 'vortex lattice', 'penetration depth']
extra_v = np.asarray(client.embed(extra), dtype=np.float32)
extra_v /= np.linalg.norm(extra_v, axis=1, keepdims=True).clip(min=1e-9)
augmented = np.vstack([route_vectors['wikipedia-superconductors'], extra_v])
for q in ['What is a penetration depth?', 'How are SQUID magnetometers used?']:
    qv = np.asarray(client.embed([q])[0], dtype=np.float32)
    qv /= np.linalg.norm(qv) + 1e-9
    before = float((route_vectors['wikipedia-superconductors'] @ qv).max())
    after = float((augmented @ qv).max())
    print(f'  {q[:50]:50s}  before={before:.3f}  after={after:.3f}')
extra = ['superconducting magnet', 'persistent current', 'flux quantum', 'vortex lattice', 'penetration depth']
extra_v = np.asarray(client.embed(extra), dtype=np.float32)
extra_v /= np.linalg.norm(extra_v, axis=1, keepdims=True).clip(min=1e-9)
augmented = np.vstack([route_vectors['wikipedia-superconductors'], extra_v])
for q in ['What is a penetration depth?', 'How are SQUID magnetometers used?']:
    qv = np.asarray(client.embed([q])[0], dtype=np.float32)
    qv /= np.linalg.norm(qv) + 1e-9
    before = float((route_vectors['wikipedia-superconductors'] @ qv).max())
    after = float((augmented @ qv).max())
    print(f'  {q[:50]:50s}  before={before:.3f}  after={after:.3f}')

  What is a penetration depth?                        before=0.566  after=0.914
  How are SQUID magnetometers used?                   before=0.600  after=0.644

Run It¶

End-to-end query with router.

In [11]:

Copied!





for q in [
    'What is selective scan in state-space models?',
    'How does flux pinning work in superconductors?',
    'What is the Apollo platform from Palantir?',
    'How do lifetimes work in Rust?',
]:
    ans, _ = answer_question(q)
    print(f'Q: {q}')
    print(f'  -> {ans[:200]}')
    print()
for q in [
    'What is selective scan in state-space models?',
    'How does flux pinning work in superconductors?',
    'What is the Apollo platform from Palantir?',
    'How do lifetimes work in Rust?',
]:
    ans, _ = answer_question(q)
    print(f'Q: {q}')
    print(f'  -> {ans[:200]}')
    print()

Q: What is selective scan in state-space models?
  -> The passages do not contain an explanation of "selective scan" in state-space models. They do mention a "selective mechanism" in the context of the Mamba model, but they do not provide a definition or

Q: How does flux pinning work in superconductors?
  -> Flux pinning works by preventing flux vortices in a type-II superconductor from moving within the bulk of the superconductor, effectively "pinning" the magnetic field lines to specific locations. This

Q: What is the Apollo platform from Palantir?
  -> The Apollo platform from Palantir is a cloud-agnostic, single control layer that coordinates ongoing delivery of new features, security updates, and platform configurations, helping to ensure the cont

Q: How do lifetimes work in Rust?
  -> The passages provided do not contain a detailed explanation of how lifetimes work in Rust. However, they do mention that lifetimes are "a variety of generics that give the compiler information about h

Side by Side: Vanilla Baseline vs This Technique¶

Vanilla (single fixed corpus) vs routed (picks the right corpus). Vanilla only knows one corpus; routed dispatches.

In [12]:

Copied!





from cookbook.baselines import vanilla_pipeline

q = 'What is the Meissner effect?'
base = vanilla_pipeline(q, corpus='rust-book', top_k=5)   # wrong corpus by design
ours_a, _ = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla (rust-book only)', 'preview': base.answer[:160]},
    {'pipeline': 'semantic router', 'preview': ours_a[:160]},
])
from cookbook.baselines import vanilla_pipeline

q = 'What is the Meissner effect?'
base = vanilla_pipeline(q, corpus='rust-book', top_k=5)   # wrong corpus by design
ours_a, _ = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla (rust-book only)', 'preview': base.answer[:160]},
    {'pipeline': 'semantic router', 'preview': ours_a[:160]},
])

Out[12]:

	pipeline	preview
0	vanilla (rust-book only)	The passages do not contain the answer. There ...
1	semantic router	The Meissner effect is the expulsion of a magn...

Knobs to Turn¶

Five knobs:

Example phrases. More and more diverse improves routing. Tune by reading the misclassifications and adding examples that cover the gap.
Threshold. Default 0.4. Lower means more aggressive routing (more false positives); higher means more refusals (more false negatives).
Aggregation across examples. Max is the cookbook default. Mean is smoother but less responsive to single matching phrases.
Use centroid vectors. Average each route's example vectors and store the centroid. Faster at query time; slightly less accurate than max-over-examples.
Compose with LLM-classifier fall-through. When the router score is low, escalate to an LLM-classifier (Recipe 26). Best of both.

Evaluate on a Slice¶

Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.

In [13]:

Copied!





from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)

Out[13]:

	question	expected	actual	contexts_retrieved
0	What problem do state-space models aim to solv...	State-space models target the quadratic time a...	The passages do not contain a direct compariso...	5
1	Describe the selective scan mechanism introduc...	Selective scan makes the SSM parameters input-...	The passages do not contain a detailed descrip...	5
2	How does Mamba achieve hardware efficiency on ...	Mamba uses a parallel scan implementation with...	The passages do not contain the answer. They m...	5
3	Which earlier model family does Mamba descend ...	Mamba builds on the structured state-space seq...	The passages do not contain the answer.	5
4	Name two domains beyond text where SSM-style b...	Audio modeling and genomics have both seen suc...	The passages do not contain the answer to the ...	5

Closing Thoughts¶

Three failure modes:

Overlapping routes. Two routes that share vocabulary cause mis-routes. Re-examine route boundaries and add disambiguating examples.
Out-of-distribution drift. New topic categories appear in user queries that don't match any route. Set up a fall-through telemetry channel — log low-score queries and use them to seed new routes.
Threshold tuning. Too high refuses good queries; too low routes bad queries to wrong indexes. Sweep on a labelled slice.

Compose with Adaptive-RAG (Recipe 26) for LLM-classifier fall-through. Compose with document-summary routing (Recipe 11) for a two-level router. Compose with metadata filters (Recipe 20) when chunks carry structured tags.