Hybrid Retrieval — Dense + BM25 with Reciprocal Rank Fusion¶

What problem does this solve?¶

Dense retrieval is great at paraphrase. "How does the company protect its customers' data?" finds passages about "data protection" and "information security". But dense retrieval is awful at exact-match terms — product codes, error codes, regulation citations ("15 U.S.C. § 78m"), specific dollar figures. A vector of 15 U.S.C. § 78m is just numbers and characters with no obvious semantic neighbours. BM25 is the opposite. It is a token-frequency model, so it locks onto exact matches like a guided missile. It fails on paraphrase. The pragmatic answer is to run both, fuse the rankings, and let each retriever cover the other's weakness. Reciprocal Rank Fusion is the canonical fuse: parameter-free, robust to score-scale differences, takes any two rankings and produces a single ranking.

Where it came from¶

BM25 dates to the 1990s — Robertson and Spärck Jones at Cambridge. RRF was introduced by Cormack, Clarke, and Büttcher in 2009 as a parameter-free way to combine TREC submissions. Their experiment showed that simple RRF beat learned combinations on most tasks. The hybrid dense + BM25 pattern emerged with the first wave of dense retrieval (Karpukhin et al., DPR 2020) and became standard in 2023 when LangChain and LlamaIndex shipped one-line hybrid retrievers. By 2026 every serious vector store has a native hybrid mode: Qdrant's BM42, Weaviate's BlockMax, Pinecone's hybrid index, Milvus' sparse-dense. We use a portable RRF implementation here so the technique reads clearly; production code should prefer the database's native hybrid path.

Where it fits in the RAG landscape¶

Three options for combining sparse and dense:

Reciprocal Rank Fusion (this recipe). Sum 1 / (k + rank) across both rankings. Parameter-free, robust. Default choice.
Score normalisation + linear combination. Min-max-normalise each ranking's scores, weighted-sum. Tunable but fragile when score distributions shift.
Cross-encoder reranker over the union. Take top-N from both, send all to a cross-encoder. Most accurate; slowest. Often combined with RRF as a two-stage pipeline.

Stack hybrid with reranking (Recipe 22) and you have the Anthropic contextual-retrieval recommended pipeline. Stack with contextual chunking (Recipe 7) and you have what most production systems are running by 2026.

When to use it (and when not to)¶

Use hybrid retrieval whenever your queries mix paraphrase and exact-match. SEC filings (lots of citations and codes), customer support (product SKUs), legal text (statute references), codebases (function names) — all benefit obviously. Skip hybrid only when your corpus is purely paraphrase-friendly (Wikipedia summary paragraphs) and queries never contain exact-match tokens. In practice this is rare; even Wikipedia queries often ask about specific dates or names. Skip BM25 entirely if you cannot tokenise sanely. Languages without whitespace, or domains where stemming changes meaning (chemistry names, legal text), need careful BM25 setup. A bad BM25 hurts hybrid more than no BM25.

The intuition¶

Three intuitions:

RRF turns disagreement into evidence. When dense and sparse agree on a passage, that passage scores high under both rankings, which RRF rewards heavily. When they disagree, RRF blends — neither retriever's strong but wrong picks dominate.

The constant k (default 60) matters less than people think. RRF is robust to k in the range 30–100. Pick 60 and move on; do not waste a week tuning it.

Hybrid pays off most at the bottom of the ranking. Top-1 is usually the same chunk under dense, sparse, or hybrid. The difference shows up at ranks 3–10, where hybrid pulls in chunks that one retriever ranked low and the other ranked high. That is where a downstream reranker (Recipe 22) earns its keep.

Architecture¶

flowchart LR
  Q[Query] --> D[Dense retriever
top-20]
  Q --> S[Sparse BM25
top-20]
  D --> F[Reciprocal
Rank Fusion]
  S --> F
  F --> R[Fused top-k
ready for generation]

References¶

📄 Reciprocal Rank Fusion (Cormack et al., 2009) — The original RRF paper. Two pages of math, a lifetime of utility.
📄 BM25 — Probabilistic Information Retrieval — Robertson and Zaragoza's canonical retrospective on BM25.
📄 Dense Passage Retrieval for Open-Domain QA (Karpukhin et al., 2020) — The DPR paper that popularised dense retrieval; established hybrid as a baseline.
📚 Qdrant Hybrid Search documentation — Native hybrid in our default vector store.
💻 rank-bm25 Python library — The lightweight BM25 implementation we use here.
📝 Anthropic Contextual Retrieval (uses hybrid) — Anthropic's recommended pipeline is hybrid + contextual + reranking.

Setup¶

Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.

In [1]:

Copied!





import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())

Provider: nebius  |  Chat model: meta-llama/Llama-3.3-70B-Instruct
Tracing disabled.

Build the Pipeline, Step by Step¶

Step 1 — Index Palantir's 10-K¶

The SEC filing is a perfect testbed: dense prose mixed with statute citations, dollar figures, and product names. The mix exercises both halves of the hybrid retriever.

In [2]:

Copied!





from cookbook.corpora import load_sec_10k
from cookbook.chunkers import fixed_window
from cookbook.stores import QdrantBackend

docs = list(load_sec_10k())
chunks = fixed_window(docs, target_tokens=384, overlap_tokens=48)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('hybrid', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')
from cookbook.corpora import load_sec_10k
from cookbook.chunkers import fixed_window
from cookbook.stores import QdrantBackend

docs = list(load_sec_10k())
chunks = fixed_window(docs, target_tokens=384, overlap_tokens=48)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('hybrid', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')

C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Indexed 253 chunks.

Step 2 — Build the BM25 side¶

BM25 is a token-frequency model. We tokenise each chunk by lowercasing and splitting on whitespace — naïve but fine for English prose. Production setups stem and remove stopwords; the cookbook keeps it simple so the implementation fits in one cell.

In [3]:

Copied!





from rank_bm25 import BM25Okapi

tokenised = [c.text.lower().split() for c in chunks]
bm25 = BM25Okapi(tokenised)

def bm25_search(query: str, top_k: int = 20):
    scores = bm25.get_scores(query.lower().split())
    import numpy as np
    order = np.argsort(scores)[::-1][:top_k]
    return [(int(i), float(scores[i])) for i in order]

print('Top-3 BM25 hits for "AIP commercial revenue concentration":')
for i, s in bm25_search('AIP commercial revenue concentration', top_k=3):
    print(f'  score={s:6.2f}  {chunks[i].text[:160]}')
from rank_bm25 import BM25Okapi

tokenised = [c.text.lower().split() for c in chunks]
bm25 = BM25Okapi(tokenised)

def bm25_search(query: str, top_k: int = 20):
    scores = bm25.get_scores(query.lower().split())
    import numpy as np
    order = np.argsort(scores)[::-1][:top_k]
    return [(int(i), float(scores[i])) for i in order]

print('Top-3 BM25 hits for "AIP commercial revenue concentration":')
for i, s in bm25_search('AIP commercial revenue concentration', top_k=3):
    print(f'  score={s:6.2f}  {chunks[i].text[:160]}')

Top-3 BM25 hits for "AIP commercial revenue concentration":
  score=  7.93  analyze the data they need in one place. The speed with which users can experiment and test new ideas is what makes the software stick. Data projects often fail
  score=  7.07  with commercial enterprises, who often faced fundamentally similar challenges in working with data. We have built four principal software platforms, Palantir Go
  score=  6.78  raise the barriers to entry for competition. The larger, more complex, and more technologically demanding the problem, the more likely we are to succeed. Additi

Step 3 — Build the dense side¶

Same Qdrant store we built above. Wrap it in a small function that returns (chunk_index, score) tuples so it has the same shape as the BM25 side. RRF doesn't care about scores beyond rank order, but having a consistent shape makes the code symmetric.

In [4]:

Copied!





id_to_idx = {c.chunk_id: i for i, c in enumerate(chunks)}

def dense_search(query: str, top_k: int = 20):
    qv = client.embed([query])[0]
    hits = store.search(qv, top_k=top_k)
    return [(id_to_idx[h.doc_id], float(h.score)) for h in hits]

print('Top-3 dense hits for the same query:')
for i, s in dense_search('AIP commercial revenue concentration', top_k=3):
    print(f'  score={s:.4f}  {chunks[i].text[:160]}')
id_to_idx = {c.chunk_id: i for i, c in enumerate(chunks)}

def dense_search(query: str, top_k: int = 20):
    qv = client.embed([query])[0]
    hits = store.search(qv, top_k=top_k)
    return [(id_to_idx[h.doc_id], float(h.score)) for h in hits]

print('Top-3 dense hits for the same query:')
for i, s in dense_search('AIP commercial revenue concentration', top_k=3):
    print(f'  score={s:.4f}  {chunks[i].text[:160]}')

Top-3 dense hits for the same query:

  score=0.5986  us-gaap:PerformanceSharesMember 2024-01-01 2024-12-31 0001321655 us-gaap:RestrictedStockUnitsRSUMember 2024-12-31 0001321655 us-gaap:PerformanceSharesMember 202
  score=0.5630  alongside generative AI models, including large language models (&#8220;LLMs&#8221;), directly within Gotham and/or Foundry to help operationalize AI on enterpr
  score=0.5538  us-gaap:GeographicConcentrationRiskMember us-gaap:SalesRevenueNetMember 2024-01-01 2024-12-31 0001321655 us-gaap:GeographicConcentrationRiskMember 2023-01-01 20

Step 4 — Reciprocal Rank Fusion¶

Take both rankings, for each chunk sum 1 / (k + rank). Sort. That is the whole algorithm. k=60 is the canonical constant from the Cormack paper.

In [5]:

Copied!





def rrf(rankings, k: int = 60, top_k: int = 10):
    scores: dict[int, float] = {}
    for ranking in rankings:
        for rank, (idx, _) in enumerate(ranking):
            scores[idx] = scores.get(idx, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top_k]

q = 'AIP commercial revenue concentration'
fused = rrf([dense_search(q, top_k=20), bm25_search(q, top_k=20)], top_k=5)
print('Top-5 hybrid hits:')
for idx, s in fused:
    print(f'  rrf_score={s:.4f}  {chunks[idx].text[:160]}')
def rrf(rankings, k: int = 60, top_k: int = 10):
    scores: dict[int, float] = {}
    for ranking in rankings:
        for rank, (idx, _) in enumerate(ranking):
            scores[idx] = scores.get(idx, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top_k]

q = 'AIP commercial revenue concentration'
fused = rrf([dense_search(q, top_k=20), bm25_search(q, top_k=20)], top_k=5)
print('Top-5 hybrid hits:')
for idx, s in fused:
    print(f'  rrf_score={s:.4f}  {chunks[idx].text[:160]}')

Top-5 hybrid hits:
  rrf_score=0.0320  alongside generative AI models, including large language models (&#8220;LLMs&#8221;), directly within Gotham and/or Foundry to help operationalize AI on enterpr
  rrf_score=0.0318  with commercial enterprises, who often faced fundamentally similar challenges in working with data. We have built four principal software platforms, Palantir Go
  rrf_score=0.0317  31, 2024, we had 711 customers. Our software is currently used across approximately 90 industries around the world. It is applied to a variety of use cases by u
  rrf_score=0.0306  analyze the data they need in one place. The speed with which users can experiment and test new ideas is what makes the software stick. Data projects often fail
  rrf_score=0.0290  value of the contracts continue to meet the criteria for revenue recognition, among other factors. Certain companies with which we have entered into commercial

Step 5 — Wrap as `answer_question`¶

The contract. Hybrid retrieval, then standard stuffed-context generation.

In [6]:

Copied!





PROMPT = (
    'Use only the passages below to answer the question. '
    'If they do not contain the answer, say so plainly.\n\n'
    'Passages:\n{context}\n\nQuestion: {question}\nAnswer:'
)

def answer_question(question: str, k: int = 5) -> tuple[str, list[str]]:
    fused = rrf(
        [dense_search(question, top_k=20), bm25_search(question, top_k=20)],
        top_k=k,
    )
    contexts = [chunks[i].text for i, _ in fused]
    return client.chat(PROMPT.format(context='\n\n'.join(contexts), question=question)), contexts

ans, _ = answer_question('What concentration risks does Palantir disclose for government customers?')
print(ans)
PROMPT = (
    'Use only the passages below to answer the question. '
    'If they do not contain the answer, say so plainly.\n\n'
    'Passages:\n{context}\n\nQuestion: {question}\nAnswer:'
)

def answer_question(question: str, k: int = 5) -> tuple[str, list[str]]:
    fused = rrf(
        [dense_search(question, top_k=20), bm25_search(question, top_k=20)],
        top_k=k,
    )
    contexts = [chunks[i].text for i, _ in fused]
    return client.chat(PROMPT.format(context='\n\n'.join(contexts), question=question)), contexts

ans, _ = answer_question('What concentration risks does Palantir disclose for government customers?')
print(ans)

Palantir discloses that its business could be adversely affected if it loses government customers or its ability to contract with the U.S. and other governments. The company notes that noncompliance with government contracting laws, regulations, or contractual provisions could lead to audits, internal investigations, damages, penalties, and termination of contracts, which could have a material adverse effect on its business, results of operations, financial condition, and growth prospects. Additionally, Palantir mentions that evolving government procurement policies and increased emphasis on cost over performance could also negatively impact its business. The company also notes that increased competition and bid protests in a budget-constrained environment may make it more difficult to maintain its financial performance and customer relationships, as a substantial portion of its business is awarded through competitive bidding.

Look Inside¶

Inspect — which retriever's choices dominate the fused ranking?¶

For a representative query, mark each fused-top-10 chunk with how dense and BM25 ranked it. When a chunk was top-3 for both, RRF promotes it strongly. When it was top-3 for only one, it still makes the fused top-10 if the other ranked it somewhere reasonable.

In [7]:

Copied!





q = 'AIP commercial revenue concentration'
dense = dense_search(q, top_k=20)
sparse = bm25_search(q, top_k=20)
dense_rank = {idx: r for r, (idx, _) in enumerate(dense)}
sparse_rank = {idx: r for r, (idx, _) in enumerate(sparse)}
fused = rrf([dense, sparse], top_k=10)

import pandas as pd
rows = []
for idx, s in fused:
    rows.append({
        'rrf_rank': len(rows) + 1,
        'dense_rank': dense_rank.get(idx, '>20'),
        'bm25_rank': sparse_rank.get(idx, '>20'),
        'preview': chunks[idx].text[:80],
    })
pd.DataFrame(rows)
q = 'AIP commercial revenue concentration'
dense = dense_search(q, top_k=20)
sparse = bm25_search(q, top_k=20)
dense_rank = {idx: r for r, (idx, _) in enumerate(dense)}
sparse_rank = {idx: r for r, (idx, _) in enumerate(sparse)}
fused = rrf([dense, sparse], top_k=10)

import pandas as pd
rows = []
for idx, s in fused:
    rows.append({
        'rrf_rank': len(rows) + 1,
        'dense_rank': dense_rank.get(idx, '>20'),
        'bm25_rank': sparse_rank.get(idx, '>20'),
        'preview': chunks[idx].text[:80],
    })
pd.DataFrame(rows)

Out[7]:

	rrf_rank	dense_rank	bm25_rank	preview
0	1	1	4	alongside generative AI models, including larg...
1	2	5	1	with commercial enterprises, who often faced f...
2	3	3	3	31, 2024, we had 711 customers. Our software i...
3	4	12	0	analyze the data they need in one place. The s...
4	5	7	11	value of the contracts continue to meet the cr...
5	6	6	15	1,569,605   $ 1,222,215   $ 1,071,77...
6	7	19	5	Class A common stock. 53 Table of Contents If ...
7	8	11	18	reasons outside our control, including competi...
8	9	0	>20	us-gaap:PerformanceSharesMember 2024-01-01 202...
9	10	2	>20	us-gaap:GeographicConcentrationRiskMember us-g...

Inspect — exact-match query that should favour BM25¶

A query with a precise citation or product name. BM25 should rank the exact match first; dense often misses it. Hybrid recovers the right answer in either case.

In [8]:

Copied!





q = 'Foundry'
print('Dense top-3:')
for i, s in dense_search(q, top_k=3):
    print(f'  {s:.3f}  {chunks[i].text[:140]}')
print()
print('BM25 top-3:')
for i, s in bm25_search(q, top_k=3):
    print(f'  {s:6.2f}  {chunks[i].text[:140]}')
print()
print('Hybrid top-3:')
for i, s in rrf([dense_search(q, top_k=20), bm25_search(q, top_k=20)], top_k=3):
    print(f'  {s:.4f}  {chunks[i].text[:140]}')
q = 'Foundry'
print('Dense top-3:')
for i, s in dense_search(q, top_k=3):
    print(f'  {s:.3f}  {chunks[i].text[:140]}')
print()
print('BM25 top-3:')
for i, s in bm25_search(q, top_k=3):
    print(f'  {s:6.2f}  {chunks[i].text[:140]}')
print()
print('Hybrid top-3:')
for i, s in rrf([dense_search(q, top_k=20), bm25_search(q, top_k=20)], top_k=3):
    print(f'  {s:.4f}  {chunks[i].text[:140]}')

Dense top-3:
  0.493  Results of Operations&#8212;Macroeconomic Trends . &#8221; Our Platforms We have built four principal software platforms: Gotham, Foundry, A
  0.479  pltr-20241231 0001321655 2024 FY false P10Y P6Y1M6D 462 275 275 iso4217:USD xbrli:shares iso4217:USD xbrli:shares pltr:segment xbrli:pure pl
  0.475  us-gaap:FairValueInputsLevel2Member 2024-12-31 0001321655 us-gaap:FairValueInputsLevel3Member 2024-12-31 0001321655 us-gaap:CashAndCashEquiv

BM25 top-3:
    6.11  Results of Operations&#8212;Macroeconomic Trends . &#8221; Our Platforms We have built four principal software platforms: Gotham, Foundry, A
    6.11  with commercial enterprises, who often faced fundamentally similar challenges in working with data. We have built four principal software pl
    5.60  and the accompanying notes thereto included elsewhere in this Annual Report on Form 10-K. This discussion contains forward-looking statement

Hybrid top-3:
  0.0333  Results of Operations&#8212;Macroeconomic Trends . &#8221; Our Platforms We have built four principal software platforms: Gotham, Foundry, A
  0.0309  with commercial enterprises, who often faced fundamentally similar challenges in working with data. We have built four principal software pl
  0.0308  alongside generative AI models, including large language models (&#8220;LLMs&#8221;), directly within Gotham and/or Foundry to help operatio

Inspect — paraphrase query that should favour dense¶

The opposite test: a query that uses none of the exact words in the answer. BM25 will miss; dense should land it.

In [9]:

Copied!





q = 'How does the company keep customer data safe from unauthorised access?'
print('Dense top-3:')
for i, s in dense_search(q, top_k=3):
    print(f'  {s:.3f}  {chunks[i].text[:140]}')
print()
print('BM25 top-3:')
for i, s in bm25_search(q, top_k=3):
    print(f'  {s:6.2f}  {chunks[i].text[:140]}')
print()
print('Hybrid top-3:')
for i, s in rrf([dense_search(q, top_k=20), bm25_search(q, top_k=20)], top_k=3):
    print(f'  {s:.4f}  {chunks[i].text[:140]}')
q = 'How does the company keep customer data safe from unauthorised access?'
print('Dense top-3:')
for i, s in dense_search(q, top_k=3):
    print(f'  {s:.3f}  {chunks[i].text[:140]}')
print()
print('BM25 top-3:')
for i, s in bm25_search(q, top_k=3):
    print(f'  {s:6.2f}  {chunks[i].text[:140]}')
print()
print('Hybrid top-3:')
for i, s in rrf([dense_search(q, top_k=20), bm25_search(q, top_k=20)], top_k=3):
    print(f'  {s:.4f}  {chunks[i].text[:140]}')

Dense top-3:
  0.575  organization is subject to unique requirements and concerns. The ways in which these principles are realized will differ among products and 
  0.552  similar security breaches and incidents, unauthorized tampering, bad actors, or human error. If an actual or perceived breach of security me
  0.545  across the full enterprise. Research and Development We believe that in order to fully address the most complex and valuable challenges that

BM25 top-3:
   11.80  analyze the data they need in one place. The speed with which users can experiment and test new ideas is what makes the software stick. Data
   11.39  of the critical audit matter does not alter in any way our opinion on the consolidated financial statements, taken as a whole, and we are no
    9.05  the Company holds at least a 20% ownership interest and has the ability to exercise significant influence over, but does not control, the in

Hybrid top-3:
  0.0262  reputation or competitive position. The natural sunsetting or phasing out of third-party products and operating systems that we use requires
  0.0167  organization is subject to unique requirements and concerns. The ways in which these principles are realized will differ among products and 
  0.0167  analyze the data they need in one place. The speed with which users can experiment and test new ideas is what makes the software stick. Data

Inspect — how RRF's `k` constant changes the ranking¶

Sweep k from small (10) to large (200). The top-5 usually does not move; the bottom of the top-20 shifts. This is why RRF is considered parameter-free — the constant matters at the tail, not at the head.

In [10]:

Copied!





q = 'AIP commercial revenue concentration'
d = dense_search(q, top_k=20); s = bm25_search(q, top_k=20)
for k_val in (10, 30, 60, 100, 200):
    top5 = [idx for idx, _ in rrf([d, s], k=k_val, top_k=5)]
    print(f'  k={k_val:3d}  top-5 ids: {top5}')
q = 'AIP commercial revenue concentration'
d = dense_search(q, top_k=20); s = bm25_search(q, top_k=20)
for k_val in (10, 30, 60, 100, 200):
    top5 = [idx for idx, _ in rrf([d, s], k=k_val, top_k=5)]
    print(f'  k={k_val:3d}  top-5 ids: {top5}')

  k= 10  top-5 ids: [165, 10, 13, 12, 73]
  k= 30  top-5 ids: [165, 10, 13, 12, 73]
  k= 60  top-5 ids: [165, 10, 13, 12, 73]
  k=100  top-5 ids: [165, 10, 13, 12, 73]
  k=200  top-5 ids: [165, 10, 13, 12, 73]

Run It¶

End-to-end on a question that needs both halves of hybrid: government-concentration risk wording (paraphrase) plus specific revenue language (exact-match).

In [11]:

Copied!





q = 'What does the company say about concentration risks tied to specific government customers and how is AIP positioned in the commercial segment?'
ans, ctxs = answer_question(q)
print('=== Hybrid answer ===')
print(ans)
print()
print(f'(used {len(ctxs)} contexts)')
q = 'What does the company say about concentration risks tied to specific government customers and how is AIP positioned in the commercial segment?'
ans, ctxs = answer_question(q)
print('=== Hybrid answer ===')
print(ans)
print()
print(f'(used {len(ctxs)} contexts)')

=== Hybrid answer ===
The company mentions that a substantial portion of its business is awarded through competitive bidding and that large government customers are subject to uncertainties regarding budgets and spending levels, which can make it difficult to predict sales. However, the company does not specifically discuss concentration risks tied to specific government customers.

Regarding the commercial segment, the company states that 45% of its revenue comes from commercial customers and that it is focused on building strategic relationships with customers in both the commercial and government sectors. The company believes that its proximity to businesses in various industries has enhanced its product and business development efforts, and it anticipates that its reach among commercial customers will accelerate moving forward. Additionally, the company notes that its customer acquisition strategy targets large-scale opportunities at commercial institutions, and it is positioned to succeed in complex and technologically demanding projects.

(used 5 contexts)

Side by Side: Vanilla Baseline vs This Technique¶

Vanilla baseline vs hybrid. Vanilla uses dense alone; hybrid adds BM25. The interesting case is a question with both paraphrase and exact-match components.

In [12]:

Copied!





from cookbook.baselines import vanilla_pipeline

q = 'What does the company say about concentration risks tied to specific government customers and how is AIP positioned in the commercial segment?'
base = vanilla_pipeline(q, corpus='sec-10k-pltr', top_k=5)
ours_a, ours_c = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla (dense only)', 'top_preview': base.contexts[0][:140]},
    {'pipeline': 'hybrid (dense + bm25)', 'top_preview': ours_c[0][:140]},
])
from cookbook.baselines import vanilla_pipeline

q = 'What does the company say about concentration risks tied to specific government customers and how is AIP positioned in the commercial segment?'
base = vanilla_pipeline(q, corpus='sec-10k-pltr', top_k=5)
ours_a, ours_c = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla (dense only)', 'top_preview': base.contexts[0][:140]},
    {'pipeline': 'hybrid (dense + bm25)', 'top_preview': ours_c[0][:140]},
])

Out[12]:

	pipeline	top_preview
0	vanilla (dense only)	customers during the trailing twelve months en...
1	hybrid (dense + bm25)	our business could cause us to lose government...

Knobs to Turn¶

Five knobs:

Top-N per retriever. We use top-20 for both before fusing. Larger N catches more recall; the cost is the BM25 scoring loop (cheap) and the dense retrieval (cheap too). Production systems often go to 50 or 100.
RRF k constant. Default 60. Lower k rewards top-rank agreement more sharply; higher k flattens the curve. Most sweeps find the result is insensitive to k in [30, 100].
Tokeniser for BM25. Whitespace split is the cookbook default. For multi-word entities ("New York") or stemmed forms, use a smarter tokeniser. Snowball stemming usually adds 1–3 points on technical corpora.
Weight on the dense ranking. Vanilla RRF treats both rankings equally. Weighted RRF gives one ranking more influence; useful when you know your corpus favours one side. Tune with a held-out eval.
Native hybrid in the database. Qdrant BM42, Weaviate BlockMax, Pinecone hybrid. The performance and ergonomic wins are real at scale. The cookbook code uses portable RRF for clarity; production code should use the native path.

Evaluate on a Slice¶

Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.

In [13]:

Copied!





from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'sec-10k-pltr']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'sec-10k-pltr']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)

Out[13]:

	question	expected	actual	contexts_retrieved
0	What are Palantir's two principal software pla...	Palantir markets Gotham and Foundry, with Apol...	The passages do not mention that Palantir has ...	5
1	What is the stated mission of Palantir accordi...	Palantir's stated mission is to make instituti...	The passages provided do not contain a stateme...	5
2	What customer segments does Palantir distinguish?	Government customers and commercial customers,...	Palantir distinguishes two customer segments: ...	5
3	What is AIP, as described in the filing?	The Artificial Intelligence Platform, Palantir...	AIP, or Artificial Intelligence Platform, is a...	5
4	Name one risk factor Palantir highlights relat...	Concentration with a small number of governmen...	One risk factor Palantir highlights related to...	5

Closing Thoughts¶

Three failure modes:

Bad BM25 tokenisation. Whitespace splitting misses casing, multi-word terms, and hyphens. The cookbook does not bother to fix this because the corpus is simple; production must.
One retriever dominates the union. If your dense scores are well-calibrated and BM25 is noisy, the fused ranking essentially equals the dense ranking. Either narrow the BM25 source corpus (per-section BM25) or weight the dense side down.
Score scale leakage. If you ever sum raw scores from dense and BM25 (instead of using RRF), score-scale differences will destroy your ranking. Stick to RRF; do not try to be clever with score arithmetic.

Hybrid is the cheapest single retrieval upgrade in this cookbook. Anthropic's contextual retrieval pipeline (contextual chunks + BM25 + reranker) is built on it. Add cross-encoder reranking (Recipe 22) on top and you have most of what production systems use in 2026.