The Embedding Zoo — Picking the Right Model¶
What problem does this solve?¶
Embedding choice is the single highest-leverage decision in a RAG pipeline. A weak model puts the right chunk at rank 8 instead of rank 1; no amount of clever reranking recovers from that completely. A strong model on the wrong domain is just as bad — a multilingual general-purpose embedder is worse than a small code-specific embedder when the corpus is software documentation. Most teams skip the comparison and pick whatever was top of the MTEB leaderboard the week they started. That is a fine first move, but the leaderboard does not know your corpus. This notebook shows you how to run a small, honest A/B between two or three embedders on your own data with your own questions.
Where it came from¶
The MTEB benchmark (Muennighoff et al., 2022) put 8 task families and 56 datasets behind a single leaderboard and made embedding choice tractable for the first time. The 2024–2026 wave — Voyage, Cohere Embed 4, BGE-M3, Qwen3-Embedding-8B, Nomic Embed v2 — pushed open-weight models within striking distance of the hosted leaders, and turned a $0.10/M-tokens decision into a $0.02 decision for most workloads.
Where it fits in the RAG landscape¶
Three families to know in 2026:
- Hosted leaders — Voyage 3 Large, Cohere Embed 4. Best out-of-the-box quality on English prose. Pay-per-token; lock-in is moderate.
- Open-weight leaders — BGE-M3, Qwen3-Embedding-8B, Nomic Embed v2. Run on your own GPU or via Nebius. Comparable to hosted leaders on standard tasks; weaker on long context unless you specifically tested for it.
- Specialised — code embeddings (Voyage Code), multimodal (Cohere Embed 4 Multimodal, Nomic Embed Vision), Matryoshka-trained models that let you slice the leading dimensions for cheap two-stage retrieval (Recipe 12).
When to use it (and when not to)¶
Run this notebook before you commit to a model. Pick a representative slice of your corpus, write down 8–12 honest evaluation questions, and measure recall@5 across the candidates. The minute you change embedding family, you have to re-embed your entire corpus — so the cost of switching later is high, and the value of getting it right once is correspondingly high. Skip the comparison only if you are still in throwaway-prototype mode. Production systems should know exactly why they picked their embedder.
The intuition¶
Three intuitions that come up over and over:
Domain trumps benchmark. A model that scores 4 points lower on MTEB but was pretrained on your domain will usually beat a leaderboard champion on your data. Code corpora love code embeddings; medical corpora love clinical embeddings.
Long context matters more than people realise. A 512-token window means you have to over-chunk dense documents and the chunks miss surrounding context. Most modern embedders support 8K+; a few stretch to 32K. If your documents are long, this is the second axis to check after quality.
Cost differences are real but small at hobbyist scale. Voyage 3 Large is roughly 3x the per-token cost of BGE-M3 via Nebius. For a 100K-token corpus you re-embed weekly, that is the difference between $0.10 and $0.03 per run. For 100M tokens, it is the difference between $100 and $30. Choose by quality, not by penny-pinching.
Architecture¶
flowchart LR D[Wikipedia
superconductors] --> C[Sentence-window
chunker] C --> M1[Embedder A
Qwen3-Embedding-8B] C --> M2[Embedder B
same default,
different chunks] M1 --> S1[(Qdrant A)] M2 --> S2[(Qdrant B)] Q[Eval set:
15 questions] --> M1 Q --> M2 S1 --> R1[recall@5 per model] S2 --> R1 R1 --> P[Side-by-side
table]
References¶
- 📄 MTEB: Massive Text Embedding Benchmark — The benchmark that made embedder choice tractable.
- 💻 Qwen3 Embedding model card — Open-weight model used as Nebius's default.
- 💻 BGE-M3 — One embedder, three retrieval modes — Strong multilingual baseline; native sparse + dense + multi-vector.
- 📚 Voyage 3 Embeddings — Hosted leader on English prose as of 2026.
- 📄 Matryoshka Representation Learning — Why slicing leading dimensions stays meaningful — see Recipe 12.
- 📚 LlamaIndex — choosing an embedding model — Practical guidance for swapping embedders behind the same retriever.
Setup¶
Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')
from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing
client = LLMClient()
print(f'Provider: {client.provider} | Chat model: {client.chat_model}')
print(init_tracing())
Provider: nebius | Chat model: meta-llama/Llama-3.3-70B-Instruct Tracing disabled.
Build the Pipeline, Step by Step¶
Step 1 — Load the test corpus¶
We use the Wikipedia superconductors subset because it is heterogeneous prose with named entities, formulas, and historical context — the kind of mixed content where embedder choice actually changes results. The eval set ships 20 hand-written questions on this corpus.
from cookbook.corpora import load_wikipedia_superconductors, load_eval_questions
from cookbook.chunkers import sentence_window
docs = list(load_wikipedia_superconductors())
chunks = sentence_window(docs, sentences_per_chunk=4, overlap=1)
print(f'Docs: {len(docs)}, chunks: {len(chunks)}')
qs = [q for q in load_eval_questions() if q['corpus'] == 'wikipedia-superconductors']
print(f'Eval questions on this corpus: {len(qs)}')
print(f'First question: {qs[0]["question"]}')
print(f'Expected answer prefix: {qs[0]["answer"][:120]}...')
Docs: 42, chunks: 72 Eval questions on this corpus: 20 First question: Who first observed superconductivity, and in what material? Expected answer prefix: Heike Kamerlingh Onnes observed it in mercury in 1911....
Step 2 — Define the recall@k harness¶
Given an embedder, build an index, run each eval question, check whether any of the top-5 retrieved chunks contains a token-level signature of the expected answer. The harness is intentionally permissive so it works without an LLM-as-judge; recipe 37 swaps it for RAGAS' faithfulness/relevance triad.
from cookbook.stores import QdrantBackend
def recall_at_k(model_label, embed_fn, k=5):
vectors = embed_fn([c.text for c in chunks])
store = QdrantBackend(f'zoo-{model_label}', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
hits, samples = 0, []
for q in qs[:15]:
qv = embed_fn([q['question']])[0]
retrieved = store.search(qv, top_k=k)
# cheap recall proxy: does any chunk contain a 6-char prefix
# of any 4+ char word from the gold answer?
gold_words = [w.lower() for w in q['answer'].split() if len(w) >= 4]
hit = any(any(w[:6] in r.text.lower() for w in gold_words) for r in retrieved)
if hit:
hits += 1
samples.append({'question': q['question'][:60], 'hit': hit, 'top_score': float(retrieved[0].score)})
return hits / len(qs[:15]), samples
print('Harness ready.')
Harness ready.
Step 3 — Run the default Nebius embedder¶
Establish a number for the default embedder this cookbook configures. We will compare against a second model in the next cell.
default_score, default_samples = recall_at_k('default', client.embed)
print(f'Default ({client.embed_model}): recall@5 = {default_score:.2f}')
C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
Default (Qwen/Qwen3-Embedding-8B): recall@5 = 1.00
On Wikipedia superconductors the default Qwen3-Embedding-8B usually scores in the 0.70–0.85 range with this loose recall proxy. Numbers below 0.6 suggest either a bad chunk size or a poorly-phrased eval question.
Step 4 — Run a second embedder via a different chunking strategy¶
Nebius only exposes one embedding model right now, so we cannot meaningfully A/B two models on the same key. Instead we hold the embedder fixed and change the chunking — which is the second-biggest lever. If you have an OpenAI or Voyage key, the comparison cell below shows how to swap the model.
from cookbook.chunkers import fixed_window
alt_chunks = fixed_window(docs, target_tokens=512, overlap_tokens=80)
print(f'Fixed-window chunks: {len(alt_chunks)}')
def embed_alt(texts):
return client.embed(texts)
vectors = client.embed([c.text for c in alt_chunks])
alt_store = QdrantBackend('zoo-fw', dim=len(vectors[0]))
alt_store.add([c.text for c in alt_chunks], vectors, ids=[c.chunk_id for c in alt_chunks])
alt_hits = 0
for q in qs[:15]:
qv = client.embed([q['question']])[0]
retrieved = alt_store.search(qv, top_k=5)
gold_words = [w.lower() for w in q['answer'].split() if len(w) >= 4]
if any(any(w[:6] in r.text.lower() for w in gold_words) for r in retrieved):
alt_hits += 1
alt_score = alt_hits / 15
print(f'Same embedder + 512-token fixed window: recall@5 = {alt_score:.2f}')
Fixed-window chunks: 42
Same embedder + 512-token fixed window: recall@5 = 1.00
Step 5 — Wrap the winning combination as answer_question¶
Every recipe ends by defining an answer_question(q) function so the eval slice at the bottom can compare techniques uniformly. Here it just uses whichever embedder + chunker we have already built.
best_store = store_for_default = None
# choose the better-scoring index above
vectors = client.embed([c.text for c in chunks])
best_store = QdrantBackend('zoo-best', dim=len(vectors[0]))
best_store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
def answer_question(question: str, k: int = 5) -> tuple[str, list[str]]:
qv = client.embed([question])[0]
hits = best_store.search(qv, top_k=k)
contexts = [h.text for h in hits]
answer = client.chat(
'Use only these passages.\n' + '\n\n'.join(contexts) + f'\n\nQ: {question}\nA:'
)
return answer, contexts
print('answer_question ready.')
answer_question ready.
Look Inside¶
Inspect — which questions broke?¶
Aggregate scores hide the interesting failures. We list the questions where recall@5 missed so we can read them and ask whether the embedder is to blame or the question is unfair.
import pandas as pd
df = pd.DataFrame(default_samples)
print('Missed questions:')
print(df[~df['hit']].to_string(index=False))
Missed questions: Empty DataFrame Columns: [question, hit, top_score] Index: []
Read every miss. Often the question references an entity by a name not in the corpus (a synonym, a different transliteration), or asks about a fact the corpus genuinely lacks. Both are eval-set problems, not embedder problems — fix the question, not the embedder.
Inspect — vector dimensionality and norm¶
Two sanity checks every embedder should pass. Norms close to 1 mean cosine similarity reduces to a dot product; norms far from 1 sometimes hint at a configuration issue (wrong model, missing normalisation flag).
import numpy as np
v = np.asarray(client.embed(['liquid nitrogen cooling'])[0])
print(f'dim : {v.shape[0]}')
print(f'L2 norm : {np.linalg.norm(v):.4f}')
print(f'mean : {v.mean():.4f}')
print(f'std : {v.std():.4f}')
print(f'sparsity (<1e-3) : {(np.abs(v) < 1e-3).mean():.2%}')
dim : 4096 L2 norm : 1.0000 mean : -0.0000 std : 0.0156 sparsity (<1e-3) : 6.37%
Inspect — how stable is the ranking?¶
Embed the same question twice and check the top-5 are identical. With caching on this is trivially true; with caching off, deterministic embedders should still match.
q = qs[0]['question']
vec_a = client.embed([q])[0]
vec_b = client.embed([q + ' ' + ' ' * 0])[0] # same text
import numpy as np
sim = float(np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b)))
print(f'self-cosine on the same query: {sim:.6f}')
self-cosine on the same query: 0.982744
Inspect — code snippet for an OpenAI or Voyage A/B¶
If you have an OpenAI or Voyage key, you can swap the embedder and compare in three lines. This cell prints the code rather than running it, so the notebook stays self-contained.
snippet = '''
from cookbook.providers import LLMClient
openai_client = LLMClient(provider="openai", embed_model="text-embedding-3-large")
voyage_client = LLMClient(provider="openai", embed_model="voyage-3-large") # via OpenRouter
score_openai, _ = recall_at_k("openai", openai_client.embed)
score_voyage, _ = recall_at_k("voyage", voyage_client.embed)
print(f"OpenAI text-embedding-3-large recall@5 = {score_openai:.2f}")
print(f"Voyage 3 Large recall@5 = {score_voyage:.2f}")
'''
print(snippet)
from cookbook.providers import LLMClient
openai_client = LLMClient(provider="openai", embed_model="text-embedding-3-large")
voyage_client = LLMClient(provider="openai", embed_model="voyage-3-large") # via OpenRouter
score_openai, _ = recall_at_k("openai", openai_client.embed)
score_voyage, _ = recall_at_k("voyage", voyage_client.embed)
print(f"OpenAI text-embedding-3-large recall@5 = {score_openai:.2f}")
print(f"Voyage 3 Large recall@5 = {score_voyage:.2f}")
Run It¶
Print a representative answer end-to-end so the technique that won the comparison is visible in the output, not just the number.
ans, ctxs = answer_question('What is the Meissner effect and how does it distinguish a superconductor from a perfect conductor?')
print('=== Answer ===')
print(ans)
print()
print('=== Top context ===')
print(ctxs[0][:400])
=== Answer ===
The Meissner effect is the expulsion of a magnetic field from a superconductor during its transition to the superconducting state when it is cooled below the critical temperature. This distinguishes a superconductor from a perfect conductor because, unlike a perfect conductor, a superconductor not only has zero electrical resistance, but also expels magnetic fields, causing repulsion of nearby magnets. In other words, the Meissner effect is what sets superconductors apart from perfect conductors, as it is a unique property that allows superconductors to shield themselves from magnetic fields. === Top context === # Meissner effect _Source: Wikipedia, CC BY-SA 4.0_ In condensed-matter physics, the Meissner effect is the expulsion of a magnetic field from a superconductor during its transition to the superconducting state when it is cooled below the critical temperature. This expulsion will repel a nearby magnet.
Side by Side: Vanilla Baseline vs This Technique¶
Compare the best embedder + chunker configuration against the vanilla pipeline's default settings on a representative question.
from cookbook.baselines import vanilla_pipeline
q = 'What is the Meissner effect and how does it distinguish a superconductor from a perfect conductor?'
base = vanilla_pipeline(q, corpus='wikipedia-superconductors', top_k=5)
ours_answer, ours_contexts = answer_question(q)
import pandas as pd
pd.DataFrame([
{'pipeline': 'vanilla', 'answer_preview': base.answer[:160], 'n_contexts': len(base.contexts)},
{'pipeline': 'embedding-zoo best', 'answer_preview': ours_answer[:160], 'n_contexts': len(ours_contexts)},
])
| pipeline | answer_preview | n_contexts | |
|---|---|---|---|
| 0 | vanilla | The Meissner effect is the expulsion of a magn... | 5 |
| 1 | embedding-zoo best | The Meissner effect is the expulsion of a magn... | 5 |
Knobs to Turn¶
Four levers ranked by impact:
- Embedder family. The biggest quality lever. Voyage 3 / Cohere Embed 4 / BGE-M3 / Qwen3-Embedding-8B differ by 4–10 recall points on most corpora. Swap by changing one config line; re-embed the corpus.
- Chunk size and overlap. Second biggest. Recipe 4 sweeps this systematically. Most corpora want chunks in the 256–512 token range; documents with code or tables sometimes need 800+.
- Chunking strategy. Fixed-window vs sentence-window vs semantic-boundary changes recall by 2–5 points without changing model or size. Recipe 5 covers the strategy choice.
- Number of retrieved chunks (
k). A free quality lever but each extra chunk adds tokens to the prompt. Most production systems land betweenk=3andk=8.
Re-run this comparison whenever you change embedder family, chunker, or significant chunk parameters. The infrastructure is here; the next switch is one cell away.
Evaluate on a Slice¶
Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample
qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'wikipedia-superconductors']
samples = []
for row in qs[:5]:
answer, contexts = answer_question(row['question'])
samples.append({
'question': row['question'],
'expected': row['answer'],
'actual': answer[:200],
'contexts_retrieved': len(list(contexts)),
})
import pandas as pd
pd.DataFrame(samples)
| question | expected | actual | contexts_retrieved | |
|---|---|---|---|---|
| 0 | Who first observed superconductivity, and in w... | Heike Kamerlingh Onnes observed it in mercury ... | The passages provided do not mention who first... | 5 |
| 1 | What is the Meissner effect? | The complete expulsion of magnetic flux from a... | The Meissner effect is the expulsion of a magn... | 5 |
| 2 | Distinguish Type-I from Type-II superconductors. | Type-I has a single critical field above which... | Type-I superconductors are characterized by th... | 5 |
| 3 | What does BCS theory explain? | It explains conventional superconductivity thr... | BCS theory explains many thermodynamic and ele... | 5 |
| 4 | What is a Cooper pair? | Two electrons bound together by phonon exchang... | A Cooper pair is a pair of electrons bound tog... | 5 |
Closing Thoughts¶
Three patterns you usually see when you run this honestly:
- The default is rarely the best. Whatever your stack ships with — Nebius's Qwen3, OpenAI's text-embedding-3-small, the SaaS demo's anonymous embedder — is a starting point. A 30-minute comparison against two alternatives almost always finds a 3–10 point lift.
- Recall@5 is not the whole story. A model with worse recall@5 but better recall@1 may be the right pick for a system that does not rerank. The opposite holds when you do rerank — you want a model that gets the right chunk into the top-20, even if it is at rank 12.
- Domain re-runs the picture. A model that crushed Wikipedia may be middle-of-the-pack on SEC filings. Re-run the harness whenever you switch corpus type.
When you commit to a model, write down the numbers and the date. Six months from now you will want to know whether the new top-of-the-MTEB-leaderboard model would actually beat what you have. That comparison only takes 30 minutes if you have the harness ready.