Listwise LLM Reranking — RankGPT-style Reordering¶

What problem does this solve?¶

Cross-encoder rerankers score one (query, chunk) pair at a time. They are powerful but blind to global context — they can't see what the other top-N candidates look like. When two candidates compete for the same factual role, or when one candidate's relevance only makes sense given another's absence, pointwise scoring is structurally limited. Listwise LLM reranking asks a single language model to reorder the entire candidate set in one pass. The model sees all N candidates together, can compare them, and can reason about consistency, contradiction, and complementarity. The trade-off: one LLM call per query, with prompt sizes that grow with N.

Where it came from¶

Listwise reranking via LLMs was crystallised by RankGPT (Sun et al., 2023). The original paper showed that a single GPT-3.5 call reordering 20 candidates beat cross-encoder rerankers on TREC and BEIR by 2-6 points. The technique fits the natural strengths of LLMs — global reasoning over text — and bypasses the awkwardness of training a cross-encoder for every new domain or language. By 2026 the technique has been productised. Jina Reranker v3 is listwise by default. Cohere Rerank 4 offers a listwise variant. Open implementations live in llama-index and langchain. The cookbook implements it directly so the mechanism is visible — under the hood it's just a prompt that asks the LLM to output a JSON array of indices in rank order.

Where it fits in the RAG landscape¶

Listwise reranking is the most expensive of the three reranking tiers:

Cross-encoder (Recipe 22). Pointwise; fast; per-pair scoring.
Listwise LLM (this recipe). Full-list; slow; global reasoning.
Listwise dedicated models (Jina Reranker v3). Same shape, smaller dedicated model, cheaper than full LLM.

Many production systems stack cross-encoder + listwise LLM: cross-encoder picks the top-20 from the shortlist, listwise picks the top-5 from the 20. Each tier adds quality at a known cost.

When to use it (and when not to)¶

Use listwise LLM reranking when quality justifies the cost. Research assistants, expert systems, tools where the user expects the top-1 to be obviously right. Production systems handling high-stakes queries (legal, medical, financial) often pay the listwise cost. Skip it when latency or cost matters more than the marginal quality. The technique adds 500-2000 ms per query and a real LLM bill. Skip it on very short shortlists. With 5 candidates a cross-encoder is already nearly optimal; the listwise call adds little.

The intuition¶

Four intuitions:

Global context catches contradictions. When two candidates contradict each other, the LLM can prefer the one consistent with other top candidates. Pointwise scorers can't see that consistency signal.

Prompt size grows with N. Each candidate adds chunks to the prompt. The cookbook caps candidates to 20 and chunk previews to 400 chars; production stacks may go further.

A smaller LLM is enough. The reranker doesn't need to answer — only to order. Use a fast small model (gpt-4o-mini, Llama-3.3-8B-instant, Qwen-2.5-7B). Save your frontier-tier budget for the final answer.

Listwise rerankers handle redundancy gracefully. If three candidates are near-duplicates, the LLM tends to rank one near the top and demote the others. Pointwise scorers would rank all three high and waste top-k slots.

Architecture¶

flowchart LR
  Q[Query] --> S[Shortlist
top-N candidates]
  S --> P[LLM:
read all N,
output order JSON]
  P --> R[Reordered top-k]
  R --> G[Generator]

References¶

📄 Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent (Sun et al., 2023) — The RankGPT paper.
📝 Jina Reranker v3 — listwise multilingual — Production listwise reranker.
📚 LlamaIndex RankGPTRerank — Reference implementation.
📚 Cohere Rerank documentation — The pointwise leader; some endpoints offer listwise.
📚 Cross-encoder rerank (Recipe 22) — The faster cousin.
📄 MMR (Recipe 21) — Diversity-aware alternative.

Setup¶

Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.

In [1]:

Copied!





import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())

Provider: nebius  |  Chat model: meta-llama/Llama-3.3-70B-Instruct
Tracing disabled.

Build the Pipeline, Step by Step¶

Step 1 — Build the index¶

Standard SEC 10-K setup. Multi-part queries are listwise reranking's home turf.

In [2]:

Copied!





from cookbook.corpora import load_sec_10k
from cookbook.chunkers import fixed_window
from cookbook.stores import QdrantBackend

docs = list(load_sec_10k())
chunks = fixed_window(docs, target_tokens=320, overlap_tokens=32)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('llm-rerank', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')
from cookbook.corpora import load_sec_10k
from cookbook.chunkers import fixed_window
from cookbook.stores import QdrantBackend

docs = list(load_sec_10k())
chunks = fixed_window(docs, target_tokens=320, overlap_tokens=32)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('llm-rerank', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')

C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Indexed 295 chunks.

Step 2 — Shortlist with the bi-encoder¶

Top-20 candidates for the listwise to reorder.

In [3]:

Copied!





q = 'What competitive risk factors does Palantir attribute to consulting firms?'
qv = client.embed([q])[0]
shortlist = store.search(qv, top_k=20)
print(f'Shortlisted {len(shortlist)} candidates.')
q = 'What competitive risk factors does Palantir attribute to consulting firms?'
qv = client.embed([q])[0]
shortlist = store.search(qv, top_k=20)
print(f'Shortlisted {len(shortlist)} candidates.')

Shortlisted 20 candidates.

Step 3 — Apply listwise LLM rerank¶

cookbook.rerankers.listwise_llm_rerank packages the prompt, parses the JSON output, and produces the reordered list.

In [4]:

Copied!





from cookbook.rerankers import listwise_llm_rerank

reranked = listwise_llm_rerank(q, shortlist, client.chat, top_k=5)
print('Listwise top-5:')
for h in reranked:
    print(f'  {h.text[:160]}')
from cookbook.rerankers import listwise_llm_rerank

reranked = listwise_llm_rerank(q, shortlist, client.chat, top_k=5)
print('Listwise top-5:')
for h in reranked:
    print(f'  {h.text[:160]}')

Listwise top-5:
  on their own, they generally rely on a patchwork of custom solutions, outside consultants, IT services companies, packaged enterprise and open source software, 
  solutions or in-house software development projects often favored by internal IT departments or other competitive products and services. In addition, our compet
  business and results of operations may be adversely affected. We believe that maintaining and enhancing our brand identity and reputation is important to our re
  in Delaware on May&#160;6, 2003. The Company builds and deploys software platforms that serve as the central operating systems for its customers. 2. Significant
  compliance obligations while considering the context of specific workflows. 9 Table of Contents &#8226; Sensitive Data Discovery and Management. Palantir&#8217;

Step 4 — Compare bi-encoder, cross-encoder, listwise¶

Three rankings, one chart. The orderings should differ in interesting ways — listwise often promotes chunks that are complementary to the top-1 rather than redundant with it.

In [5]:

Copied!





from cookbook.rerankers import cross_encoder_rerank

be_ids = [h.doc_id for h in shortlist[:5]]
ce_ids = [h.doc_id for h in cross_encoder_rerank(q, shortlist, top_k=5, chat=client.chat)]
lw_ids = [h.doc_id for h in listwise_llm_rerank(q, shortlist, client.chat, top_k=5)]

import pandas as pd
rows = [
    {'rank': r+1, 'bi_encoder': be_ids[r][:20] if r < len(be_ids) else None,
     'cross_encoder': ce_ids[r][:20] if r < len(ce_ids) else None,
     'listwise_llm': lw_ids[r][:20] if r < len(lw_ids) else None}
    for r in range(5)
]
pd.DataFrame(rows)
from cookbook.rerankers import cross_encoder_rerank

be_ids = [h.doc_id for h in shortlist[:5]]
ce_ids = [h.doc_id for h in cross_encoder_rerank(q, shortlist, top_k=5, chat=client.chat)]
lw_ids = [h.doc_id for h in listwise_llm_rerank(q, shortlist, client.chat, top_k=5)]

import pandas as pd
rows = [
    {'rank': r+1, 'bi_encoder': be_ids[r][:20] if r < len(be_ids) else None,
     'cross_encoder': ce_ids[r][:20] if r < len(ce_ids) else None,
     'listwise_llm': lw_ids[r][:20] if r < len(lw_ids) else None}
    for r in range(5)
]
pd.DataFrame(rows)

Out[5]:

	rank	bi_encoder	cross_encoder	listwise_llm
0	1	sec:PLTR-2024-10K#al	sec:PLTR-2024-10K#al	sec:PLTR-2024-10K#al
1	2	sec:PLTR-2024-10K#al	sec:PLTR-2024-10K#al	sec:PLTR-2024-10K#al
2	3	sec:PLTR-2024-10K#al	sec:PLTR-2024-10K#al	sec:PLTR-2024-10K#al
3	4	sec:PLTR-2024-10K#al	sec:PLTR-2024-10K#al	sec:PLTR-2024-10K#al
4	5	sec:PLTR-2024-10K#al	sec:PLTR-2024-10K#al	sec:PLTR-2024-10K#al

Step 5 — Wrap as `answer_question`¶

Cookbook contract.

In [6]:

Copied!





PROMPT = (
    'Use only the passages below to answer the question.\n\n'
    'Passages:\n{context}\n\nQuestion: {question}\nAnswer:'
)

def answer_question(question: str, k: int = 5) -> tuple[str, list[str]]:
    qv = client.embed([question])[0]
    short = store.search(qv, top_k=20)
    top = listwise_llm_rerank(question, short, client.chat, top_k=k)
    contexts = [h.text for h in top]
    return client.chat(PROMPT.format(context='\n\n'.join(contexts), question=question)), contexts

ans, _ = answer_question('What customer concentration metrics does Palantir disclose?')
print(ans)
PROMPT = (
    'Use only the passages below to answer the question.\n\n'
    'Passages:\n{context}\n\nQuestion: {question}\nAnswer:'
)

def answer_question(question: str, k: int = 5) -> tuple[str, list[str]]:
    qv = client.embed([question])[0]
    short = store.search(qv, top_k=20)
    top = listwise_llm_rerank(question, short, client.chat, top_k=k)
    contexts = [h.text for h in top]
    return client.chat(PROMPT.format(context='\n\n'.join(contexts), question=question)), contexts

ans, _ = answer_question('What customer concentration metrics does Palantir disclose?')
print(ans)

Palantir discloses the following customer concentration metrics:

1. As of December 31, 2024, and 2023, one customer, referred to as "Customer I", represented 26% and 15%, respectively, of total accounts receivable.
2. No other customer represented more than 10% of total accounts receivable as of December 31, 2024, or 2023.
3. For the years ended December 31, 2024, 2023, and 2022, no customer represented 10% or more of total revenue.

Look Inside¶

Inspect — latency comparison¶

Listwise costs one LLM call per query. Cross-encoder costs N pair scorings. Bi-encoder is essentially free. Measure.

In [7]:

Copied!





import time
from cookbook.rerankers import cross_encoder_rerank

t0 = time.perf_counter()
_ = store.search(qv, top_k=20)
be_ms = (time.perf_counter() - t0) * 1000

t0 = time.perf_counter()
_ = cross_encoder_rerank(q, shortlist, top_k=5, chat=client.chat)
ce_ms = (time.perf_counter() - t0) * 1000

t0 = time.perf_counter()
_ = listwise_llm_rerank(q, shortlist, client.chat, top_k=5)
lw_ms = (time.perf_counter() - t0) * 1000

print(f'Bi-encoder retrieval : {be_ms:7.1f} ms')
print(f'Cross-encoder rerank : {ce_ms:7.1f} ms')
print(f'Listwise LLM rerank  : {lw_ms:7.1f} ms')
import time
from cookbook.rerankers import cross_encoder_rerank

t0 = time.perf_counter()
_ = store.search(qv, top_k=20)
be_ms = (time.perf_counter() - t0) * 1000

t0 = time.perf_counter()
_ = cross_encoder_rerank(q, shortlist, top_k=5, chat=client.chat)
ce_ms = (time.perf_counter() - t0) * 1000

t0 = time.perf_counter()
_ = listwise_llm_rerank(q, shortlist, client.chat, top_k=5)
lw_ms = (time.perf_counter() - t0) * 1000

print(f'Bi-encoder retrieval : {be_ms:7.1f} ms')
print(f'Cross-encoder rerank : {ce_ms:7.1f} ms')
print(f'Listwise LLM rerank  : {lw_ms:7.1f} ms')

Bi-encoder retrieval :    45.2 ms
Cross-encoder rerank :    18.9 ms
Listwise LLM rerank  :     1.4 ms

Inspect — does listwise change the top-1?¶

Run several queries and count how often listwise picks a different top-1 than cross-encoder.

In [8]:

Copied!





battery = [
    'What competitive risks does Palantir disclose?',
    'How does Palantir generate revenue from AIP?',
    'What governance structures protect founders?',
    'What cybersecurity controls does the filing describe?',
]
same, diff = 0, 0
for q in battery:
    qv = client.embed([q])[0]
    short = store.search(qv, top_k=20)
    ce_top = cross_encoder_rerank(q, short, top_k=1, chat=client.chat)[0]
    lw_top = listwise_llm_rerank(q, short, client.chat, top_k=1)[0]
    if ce_top.doc_id == lw_top.doc_id:
        same += 1
    else:
        diff += 1
print(f'Same top-1: {same}/{len(battery)}; Different: {diff}/{len(battery)}')
battery = [
    'What competitive risks does Palantir disclose?',
    'How does Palantir generate revenue from AIP?',
    'What governance structures protect founders?',
    'What cybersecurity controls does the filing describe?',
]
same, diff = 0, 0
for q in battery:
    qv = client.embed([q])[0]
    short = store.search(qv, top_k=20)
    ce_top = cross_encoder_rerank(q, short, top_k=1, chat=client.chat)[0]
    lw_top = listwise_llm_rerank(q, short, client.chat, top_k=1)[0]
    if ce_top.doc_id == lw_top.doc_id:
        same += 1
    else:
        diff += 1
print(f'Same top-1: {same}/{len(battery)}; Different: {diff}/{len(battery)}')

Same top-1: 2/4; Different: 2/4

Inspect — what does the LLM prompt look like?¶

Read the prompt we send to the LLM. It's just a numbered list of passages plus a directive.

In [9]:

Copied!





rendered = '\n'.join(f'[{i}] {h.text[:200]}' for i, h in enumerate(shortlist[:10]))
preview = (
    'Reorder the following passages from most to least relevant for the query.\n'
    f'Query: {q}\n\nPassages:\n{rendered}\n\n'
    'Respond as JSON: {"order": [3, 1, 7, ...]} with passage indices.'
)
print(preview[:1200])
print('...')
rendered = '\n'.join(f'[{i}] {h.text[:200]}' for i, h in enumerate(shortlist[:10]))
preview = (
    'Reorder the following passages from most to least relevant for the query.\n'
    f'Query: {q}\n\nPassages:\n{rendered}\n\n'
    'Respond as JSON: {"order": [3, 1, 7, ...]} with passage indices.'
)
print(preview[:1200])
print('...')

Reorder the following passages from most to least relevant for the query.
Query: What cybersecurity controls does the filing describe?

Passages:
[0] in Delaware on May&#160;6, 2003. The Company builds and deploys software platforms that serve as the central operating systems for its customers. 2. Significant Accounting Policies Basis of Presentati
[1] compliance obligations while considering the context of specific workflows. 9 Table of Contents &#8226; Sensitive Data Discovery and Management. Palantir&#8217;s software platforms enable users to sec
[2] have not experienced any work stoppages due to employee disputes, and we believe that our employee relations are strong. Our human capital resources objectives include recruiting, retaining, training,
[3] to host or operate some or all of certain key technology platform features or functions of our business, including our cloud-based services (including Palantir Cloud, as defined in the section titled 
[4] technology industry overall has increased and we have engaged more actively with media and marketing efforts, we have attracted, and may continue to attract, significant attention from news and social
[5] recorded as necessary to 
...

Inspect — cost¶

Listwise adds one full LLM call. Cache helps but only on repeated queries.

In [10]:

Copied!





from cookbook import _cache
before = _cache.stats()['entries']
_ = answer_question('What does the company say about consulting competitors?')
after = _cache.stats()['entries']
print(f'New cache entries: {after - before}')
print('Rough breakdown:')
print('  1   query embed')
print('  1   listwise rerank LLM call')
print('  1   final-answer LLM call')
from cookbook import _cache
before = _cache.stats()['entries']
_ = answer_question('What does the company say about consulting competitors?')
after = _cache.stats()['entries']
print(f'New cache entries: {after - before}')
print('Rough breakdown:')
print('  1   query embed')
print('  1   listwise rerank LLM call')
print('  1   final-answer LLM call')

New cache entries: 0
Rough breakdown:
  1   query embed
  1   listwise rerank LLM call
  1   final-answer LLM call

Run It¶

End-to-end on a representative SEC question.

In [11]:

Copied!





q = 'What does Palantir say about its competition with consulting firms and large platform companies?'
ans, _ = answer_question(q)
print('=== Listwise-reranked answer ===')
print(ans)
q = 'What does Palantir say about its competition with consulting firms and large platform companies?'
ans, _ = answer_question(q)
print('=== Listwise-reranked answer ===')
print(ans)

=== Listwise-reranked answer ===
According to the passages, Palantir competes with a range of companies, including large enterprise software companies, government contractors, system integrators, and emerging companies. The company notes that some of its competitors have substantial competitive advantages, such as greater name recognition, larger customer bases, and more resources. Palantir also competes with consulting firms and IT services companies, as potential customers may attempt to build their own data platforms using a patchwork of custom solutions, outside consultants, and packaged enterprise and open source software. Additionally, Palantir faces competition from larger platform companies that may offer bundled products or services, and may have greater resources and broader product lines. The company acknowledges that it must differentiate its platforms from those of its competitors in order to succeed, and that failure to do so could result in decreased demand and adverse effects on its business and financial condition.

Side by Side: Vanilla Baseline vs This Technique¶

Vanilla vs listwise.

In [12]:

Copied!





from cookbook.baselines import vanilla_pipeline

q = 'What does Palantir say about its competition with consulting firms and large platform companies?'
base = vanilla_pipeline(q, corpus='sec-10k-pltr', top_k=5)
ours_a, _ = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'preview': base.answer[:160]},
    {'pipeline': 'listwise-llm', 'preview': ours_a[:160]},
])
from cookbook.baselines import vanilla_pipeline

q = 'What does Palantir say about its competition with consulting firms and large platform companies?'
base = vanilla_pipeline(q, corpus='sec-10k-pltr', top_k=5)
ours_a, _ = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'preview': base.answer[:160]},
    {'pipeline': 'listwise-llm', 'preview': ours_a[:160]},
])

Out[12]:

	pipeline	preview
0	vanilla	The passages do not contain information about ...
1	listwise-llm	According to the passages, Palantir competes w...

Knobs to Turn¶

Six knobs in priority order:

Shortlist size. Default 20. Higher gives the LLM more material but inflates prompt length and per-query cost.
Reranker model. A small fast model is fine. Reserve your best model for the final answer — the reranker only needs to order, not to answer.
Prompt structure. RankGPT-style numbered passages plus JSON output. The cookbook uses this format; it's robust across providers.
Snippet length per passage. We use 400 chars. Longer gives the LLM more to compare; shorter keeps the prompt small and the LLM call cheap.
Composition. Stack on top of cross-encoder rerank: cross-encoder picks top-20 from 100, listwise picks top-5 from 20.
JSON fallback. Always have a fallback for malformed JSON. The cookbook returns the original ordering on parse failure.

Evaluate on a Slice¶

Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.

In [13]:

Copied!





from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'sec-10k-pltr']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'sec-10k-pltr']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)

Out[13]:

	question	expected	actual	contexts_retrieved
0	What are Palantir's two principal software pla...	Palantir markets Gotham and Foundry, with Apol...	The passages do not explicitly state the names...	5
1	What is the stated mission of Palantir accordi...	Palantir's stated mission is to make instituti...	The stated mission of Palantir is not explicit...	5
2	What customer segments does Palantir distinguish?	Government customers and commercial customers,...	Palantir distinguishes two customer segments: ...	5
3	What is AIP, as described in the filing?	The Artificial Intelligence Platform, Palantir...	AIP, or Artificial Intelligence Platform, is a...	5
4	Name one risk factor Palantir highlights relat...	Concentration with a small number of governmen...	One risk factor Palantir highlights related to...	5

Closing Thoughts¶

Three failure modes you'll meet in production:

JSON parse failures. The LLM occasionally returns malformed JSON. Cookbook's parser falls back to the original ranking on parse failure, which is safer than refusing.
Prompt length blow-up. With N=50 and long chunks the prompt is huge, slow, and expensive. Trim snippets aggressively.
Cost. Two LLM calls per query (rerank + answer). For high-volume systems, listwise may not pay off — use a dedicated listwise reranker model like Jina v3 instead.

Compose with cross-encoder rerank (Recipe 22) for a two-tier rerank. Compose with hybrid retrieval (Recipe 18) for the shortlist. Compose with adaptive routing (Recipe 26) to only apply listwise on queries that benefit.