RAGAS Triad — Faithfulness, Relevance, Precision, Recall¶

What problem does this solve?¶

You changed the chunker, the retriever, the reranker, and the prompt — did anything actually improve? Without metrics, every change becomes vibes. "It feels better" is not a release decision. RAGAS gives you four reference-free metrics that turn the question into a number. The triad — faithfulness, answer relevance, context precision, context recall — covers the four ways a RAG pipeline can fail: hallucinating beyond the context, answering the wrong question, retrieving noise, missing crucial passages. Each metric uses an LLM-as-judge with calibrated prompts. The numbers aren't ground truth, but they're consistent enough to move with the technique under test.

Where it came from¶

RAGAS was published by ExplodingGradients in mid-2023 ("Automated Evaluation of Retrieval Augmented Generation"). The framework consolidated several earlier evaluation patterns into a single library and made them composable. By 2026 RAGAS is the default open-source evaluation tool for RAG; competitors (DeepEval, Phoenix, Langfuse) all support similar metrics with slightly different framing. The reference-free design was the key innovation. Earlier eval frameworks required ground-truth answers; RAGAS computes faithfulness and context precision from the (question, contexts, answer) triple alone. That makes it practical to run on production traffic, not just benchmarks.

Where it fits in the RAG landscape¶

Four RAG-evaluation tools to know in 2026:

RAGAS (this recipe). Reference-free metrics, mature ecosystem, integration with LlamaIndex and LangChain.
DeepEval (Recipe 38). Pytest-native metrics; better for CI integration with pass/fail thresholds.
Arize Phoenix. Traces + evaluations in one tool; visualisation-first.
Langfuse / TruLens. Production observability with evaluation; strong cost tracking.

Pick by integration: RAGAS for ad-hoc analysis, DeepEval for CI, Phoenix for tracing, Langfuse for production. The four tools overlap on metric definitions; they differ on workflow integration and visualisation. Most production stacks use at least two of them together.

When to use it (and when not to)¶

Use RAGAS when you need to measure a change. Tuning chunk size, swapping embedders, adding a reranker — RAGAS gives the move a number. Skip it for tiny prototypes. The eval overhead doesn't pay off until you're iterating. Skip it when ground-truth labels are available and you can do simple metric comparisons. Faithfulness adds value; exact-match accuracy on labelled QA is simpler.

The intuition¶

Four intuitions to carry:

The triad is four metrics. Faithfulness (no hallucination), answer relevance (answers the question), context precision (no junk retrieved), context recall (nothing important missed).

The LLM is the judge. Each metric uses a prompt to score the (question, contexts, answer) triple. The judge model matters; consistent scoring requires a consistent judge.

Numbers are relative, not absolute. RAGAS scores aren't comparable across teams or datasets. They are comparable within one team's eval set across pipeline versions.

80 questions is enough. Larger eval sets give tighter confidence intervals but rarely change the rank order of pipelines. The cookbook ships 80 questions across 4 corpora.

Architecture¶

flowchart TB
  Q[Question] --> P[RAG pipeline]
  P --> A[Answer]
  P --> C[Contexts]
  Q --> J1[Faithfulness
judge]
  A --> J1
  C --> J1
  Q --> J2[Answer relevance
judge]
  A --> J2
  Q --> J3[Context precision
judge]
  C --> J3
  Q --> J4[Context recall
judge]
  C --> J4
  GT[Ground truth] --> J4
  J1 --> M[Metric dict]
  J2 --> M
  J3 --> M
  J4 --> M

References¶

📄 RAGAS — Automated Evaluation of Retrieval Augmented Generation — The original paper.
📚 RAGAS documentation — Official docs.
💻 explodinggradients/ragas repository — Source and examples.
📚 DeepEval (Recipe 38) — The pytest-native alternative.
📚 Arize Phoenix — Tracing + evaluation tool.
📚 Langfuse evaluation — Production-observability variant.

Setup¶

Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.

In [1]:

Copied!





import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())

Provider: nebius  |  Chat model: meta-llama/Llama-3.3-70B-Instruct
Tracing disabled.

Build the Pipeline, Step by Step¶

Step 1 — Build a pipeline to evaluate¶

Standard vanilla pipeline on the Mamba paper. We'll measure this as the baseline; later recipes plug in better pipelines.

In [2]:

Copied!





from cookbook.corpora import load_arxiv_mamba, load_eval_questions
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend

docs = list(load_arxiv_mamba())
chunks = sentence_window(docs, sentences_per_chunk=4)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('ragas', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])

def answer_question(question: str) -> tuple[str, list[str]]:
    qv = client.embed([question])[0]
    hits = store.search(qv, top_k=5)
    ctx = [h.text for h in hits]
    a = client.chat('Use only these passages.\n' + '\n\n'.join(ctx) + f'\nQ: {question}\nA:')
    return a, ctx

print('Pipeline ready.')
from cookbook.corpora import load_arxiv_mamba, load_eval_questions
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend

docs = list(load_arxiv_mamba())
chunks = sentence_window(docs, sentences_per_chunk=4)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('ragas', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])

def answer_question(question: str) -> tuple[str, list[str]]:
    qv = client.embed([question])[0]
    hits = store.search(qv, top_k=5)
    ctx = [h.text for h in hits]
    a = client.chat('Use only these passages.\n' + '\n\n'.join(ctx) + f'\nQ: {question}\nA:')
    return a, ctx

print('Pipeline ready.')

C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Pipeline ready.

Step 2 — Run the pipeline on the eval slice¶

Pick 8 questions from the cookbook eval set, run the pipeline, collect (question, expected, actual, contexts).

In [3]:

Copied!





samples = []
for row in [q for q in load_eval_questions() if q['corpus'] == 'arxiv-mamba'][:8]:
    actual, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected_answer': row['answer'],
        'actual_answer': actual,
        'contexts': contexts,
    })
print(f'Collected {len(samples)} samples.')
print()
print(samples[0]['actual_answer'][:240])
samples = []
for row in [q for q in load_eval_questions() if q['corpus'] == 'arxiv-mamba'][:8]:
    actual, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected_answer': row['answer'],
        'actual_answer': actual,
        'contexts': contexts,
    })
print(f'Collected {len(samples)} samples.')
print()
print(samples[0]['actual_answer'][:240])

Collected 8 samples.

State-space models aim to solve the problem of efficient sequential modeling, particularly for long-term sequences, compared to attention-based transformers. They do this by selectively retaining relevant information from previous states, a

Step 3 — Run RAGAS metrics¶

We use cookbook.eval.evaluate, which wraps RAGAS with cookbook defaults. Each metric is computed per-sample and averaged.

In [4]:

Copied!





import os
# RAGAS uses an LLM judge — point it at Nebius via OpenAI-compatible env vars.
os.environ.setdefault('OPENAI_API_KEY', os.environ.get(client._spec.api_key_env, ''))
os.environ.setdefault('OPENAI_BASE_URL', os.environ.get(client._spec.base_url_env, '') or '')

from cookbook.eval import EvalSample, evaluate

ev_samples = [
    EvalSample(
        question=s['question'],
        expected_answer=s['expected_answer'],
        contexts=s['contexts'],
        actual_answer=s['actual_answer'],
    ) for s in samples
]
metrics = evaluate(ev_samples, use_ragas=True, use_deepeval=False)
metrics
import os
# RAGAS uses an LLM judge — point it at Nebius via OpenAI-compatible env vars.
os.environ.setdefault('OPENAI_API_KEY', os.environ.get(client._spec.api_key_env, ''))
os.environ.setdefault('OPENAI_BASE_URL', os.environ.get(client._spec.base_url_env, '') or '')

from cookbook.eval import EvalSample, evaluate

ev_samples = [
    EvalSample(
        question=s['question'],
        expected_answer=s['expected_answer'],
        contexts=s['contexts'],
        actual_answer=s['actual_answer'],
    ) for s in samples
]
metrics = evaluate(ev_samples, use_ragas=True, use_deepeval=False)
metrics

Evaluating:   0%|          | 0/32 [00:00<?, ?it/s]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:   3%|▎         | 1/32 [00:07<03:52,  7.49s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  12%|█▎        | 4/32 [00:09<00:57,  2.05s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  28%|██▊       | 9/32 [00:14<00:28,  1.25s/it]

Evaluating:  31%|███▏      | 10/32 [00:14<00:23,  1.08s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  34%|███▍      | 11/32 [00:15<00:22,  1.05s/it]

Evaluating:  38%|███▊      | 12/32 [00:16<00:20,  1.04s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  41%|████      | 13/32 [00:20<00:33,  1.76s/it]

Evaluating:  44%|████▍     | 14/32 [00:20<00:25,  1.43s/it]

Evaluating:  47%|████▋     | 15/32 [00:23<00:32,  1.90s/it]

Evaluating:  50%|█████     | 16/32 [00:26<00:34,  2.14s/it]

Evaluating:  53%|█████▎    | 17/32 [00:26<00:23,  1.59s/it]

Evaluating:  56%|█████▋    | 18/32 [00:28<00:21,  1.56s/it]

Evaluating:  59%|█████▉    | 19/32 [00:31<00:26,  2.02s/it]

Evaluating:  66%|██████▌   | 21/32 [00:31<00:12,  1.15s/it]

Evaluating:  81%|████████▏ | 26/32 [00:32<00:03,  1.85it/s]

Evaluating:  84%|████████▍ | 27/32 [00:33<00:03,  1.56it/s]

Evaluating:  88%|████████▊ | 28/32 [00:34<00:02,  1.42it/s]

Evaluating:  91%|█████████ | 29/32 [00:35<00:02,  1.28it/s]

Evaluating:  94%|█████████▍| 30/32 [00:36<00:01,  1.35it/s]

Evaluating: 100%|██████████| 32/32 [00:38<00:00,  1.30it/s]

Evaluating: 100%|██████████| 32/32 [00:38<00:00,  1.19s/it]

Out[4]:

{'ragas_faithfulness': 0.8273809523809523,
 'ragas_answer_relevancy': 0.462447530043233,
 'ragas_context_precision': 0.1187499999925,
 'ragas_context_recall': 0.125}

Step 4 — Interpret the numbers¶

Faithfulness near 1.0 means the answer is grounded; below 0.7 means the model hallucinated. Answer relevance scores how well the answer addresses the question. Context precision / recall measure whether retrieval was tight (precision) and complete (recall).

In [5]:

Copied!





import pandas as pd
rows = [{'metric': k, 'score': round(v, 3)} for k, v in metrics.items()]
df = pd.DataFrame(rows)
df['interpretation'] = df['score'].apply(lambda s: 'good' if s >= 0.7 else 'investigate' if s >= 0.5 else 'broken')
df
import pandas as pd
rows = [{'metric': k, 'score': round(v, 3)} for k, v in metrics.items()]
df = pd.DataFrame(rows)
df['interpretation'] = df['score'].apply(lambda s: 'good' if s >= 0.7 else 'investigate' if s >= 0.5 else 'broken')
df

Out[5]:

	metric	score	interpretation
0	ragas_faithfulness	0.827	good
1	ragas_answer_relevancy	0.462	broken
2	ragas_context_precision	0.119	broken
3	ragas_context_recall	0.125	broken

Step 5 — Define a function to evaluate any pipeline¶

Now you can evaluate any answer_question against the eval set. Use this template to compare pipelines from earlier recipes.

In [6]:

Copied!





def evaluate_pipeline(fn, corpus_key: str, k: int = 8):
    rows = [q for q in load_eval_questions() if q['corpus'] == corpus_key][:k]
    ev = []
    for r in rows:
        ans, ctx = fn(r['question'])
        ev.append(EvalSample(
            question=r['question'], expected_answer=r['answer'],
            contexts=ctx, actual_answer=ans,
        ))
    return evaluate(ev, use_ragas=True)

print('evaluate_pipeline ready.')
def evaluate_pipeline(fn, corpus_key: str, k: int = 8):
    rows = [q for q in load_eval_questions() if q['corpus'] == corpus_key][:k]
    ev = []
    for r in rows:
        ans, ctx = fn(r['question'])
        ev.append(EvalSample(
            question=r['question'], expected_answer=r['answer'],
            contexts=ctx, actual_answer=ans,
        ))
    return evaluate(ev, use_ragas=True)

print('evaluate_pipeline ready.')

evaluate_pipeline ready.

Look Inside¶

Inspect — read a low-scoring sample¶

When the average is low, find the worst sample and read it. Often the failure is concentrated in one or two queries.

In [7]:

Copied!





for s in samples[:3]:
    print(f'Q: {s["question"][:80]}')
    print(f'  expected: {s["expected_answer"][:120]}')
    print(f'  actual:   {s["actual_answer"][:120]}')
    print()
for s in samples[:3]:
    print(f'Q: {s["question"][:80]}')
    print(f'  expected: {s["expected_answer"][:120]}')
    print(f'  actual:   {s["actual_answer"][:120]}')
    print()

Q: What problem do state-space models aim to solve compared to attention-based tran
  expected: State-space models target the quadratic time and memory complexity of self-attention, providing linear-time sequence mod
  actual:   State-space models aim to solve the problem of efficient sequential modeling, particularly for long-term sequences, comp

Q: Describe the selective scan mechanism introduced in Mamba.
  expected: Selective scan makes the SSM parameters input-dependent so the model can choose which information to propagate or forget
  actual:   The passage does not describe the selective scan mechanism, but rather mentions a "data-dependence selection mechanism" 

Q: How does Mamba achieve hardware efficiency on modern GPUs?
  expected: Mamba uses a parallel scan implementation with kernel fusion and memory-aware layout, reaching higher throughput than eq
  actual:   RecMamba achieves hardware efficiency on modern GPUs by notably reducing GPU memory footprint and significantly slashing

Inspect — does context count matter?¶

Re-run the eval at different k values to see how chunk count affects metrics.

In [8]:

Copied!





import pandas as pd
rows = []
for k in (3, 5, 8):
    def fn_k(q, k=k):
        qv = client.embed([q])[0]
        hits = store.search(qv, top_k=k)
        ctx = [h.text for h in hits]
        a = client.chat('Use only these passages.\n' + '\n\n'.join(ctx) + f'\nQ: {q}\nA:')
        return a, ctx
    m = evaluate_pipeline(fn_k, 'arxiv-mamba', k=5)
    rows.append({'k': k, **{key.replace('ragas_', ''): round(v, 3) for key, v in m.items()}})
pd.DataFrame(rows)
import pandas as pd
rows = []
for k in (3, 5, 8):
    def fn_k(q, k=k):
        qv = client.embed([q])[0]
        hits = store.search(qv, top_k=k)
        ctx = [h.text for h in hits]
        a = client.chat('Use only these passages.\n' + '\n\n'.join(ctx) + f'\nQ: {q}\nA:')
        return a, ctx
    m = evaluate_pipeline(fn_k, 'arxiv-mamba', k=5)
    rows.append({'k': k, **{key.replace('ragas_', ''): round(v, 3) for key, v in m.items()}})
pd.DataFrame(rows)

16:13:49 - LiteLLM:WARNING: common_utils.py:979 - litellm: could not pre-load bedrock-runtime response stream shape — Bedrock event-stream decoding will be unavailable. Error: No module named 'botocore'

16:13:49 - LiteLLM:WARNING: common_utils.py:24 - litellm: could not pre-load sagemaker-runtime response stream shape — SageMaker event-stream decoding will be unavailable. Error: No module named 'botocore'

Evaluating:   0%|          | 0/20 [00:00<?, ?it/s]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:   5%|▌         | 1/20 [00:04<01:23,  4.41s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  10%|█         | 2/20 [00:07<01:00,  3.36s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  15%|█▌        | 3/20 [00:07<00:35,  2.10s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  20%|██        | 4/20 [00:08<00:23,  1.50s/it]

Evaluating:  25%|██▌       | 5/20 [00:08<00:15,  1.03s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  45%|████▌     | 9/20 [00:11<00:09,  1.15it/s]

Evaluating:  50%|█████     | 10/20 [00:11<00:07,  1.38it/s]

Evaluating:  55%|█████▌    | 11/20 [00:14<00:10,  1.19s/it]

Evaluating:  60%|██████    | 12/20 [00:14<00:07,  1.06it/s]

Evaluating:  65%|██████▌   | 13/20 [00:14<00:05,  1.35it/s]

Evaluating:  70%|███████   | 14/20 [00:16<00:05,  1.06it/s]

Evaluating:  75%|███████▌  | 15/20 [00:16<00:03,  1.30it/s]

Evaluating:  80%|████████  | 16/20 [00:16<00:02,  1.71it/s]

Evaluating:  85%|████████▌ | 17/20 [00:18<00:02,  1.25it/s]

Evaluating:  90%|█████████ | 18/20 [00:24<00:04,  2.45s/it]

Evaluating:  95%|█████████▌| 19/20 [00:30<00:03,  3.43s/it]

Evaluating: 100%|██████████| 20/20 [00:32<00:00,  3.09s/it]

Evaluating: 100%|██████████| 20/20 [00:32<00:00,  1.63s/it]

Evaluating:   0%|          | 0/20 [00:00<?, ?it/s]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:   5%|▌         | 1/20 [00:05<01:36,  5.10s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  10%|█         | 2/20 [00:10<01:32,  5.16s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  45%|████▌     | 9/20 [00:14<00:13,  1.23s/it]

Evaluating:  50%|█████     | 10/20 [00:14<00:10,  1.08s/it]

Evaluating:  55%|█████▌    | 11/20 [00:23<00:22,  2.46s/it]

Evaluating:  60%|██████    | 12/20 [00:24<00:17,  2.21s/it]

Evaluating:  75%|███████▌  | 15/20 [00:26<00:07,  1.53s/it]

Evaluating:  85%|████████▌ | 17/20 [00:27<00:03,  1.19s/it]

Evaluating: 100%|██████████| 20/20 [00:27<00:00,  1.38s/it]

Evaluating:   0%|          | 0/20 [00:00<?, ?it/s]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:   5%|▌         | 1/20 [00:05<01:51,  5.86s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  10%|█         | 2/20 [00:07<01:05,  3.63s/it]

Evaluating:  25%|██▌       | 5/20 [00:08<00:16,  1.11s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  45%|████▌     | 9/20 [00:11<00:11,  1.00s/it]

Evaluating:  50%|█████     | 10/20 [00:12<00:08,  1.15it/s]

Evaluating:  55%|█████▌    | 11/20 [00:20<00:20,  2.28s/it]

Evaluating:  60%|██████    | 12/20 [00:20<00:14,  1.82s/it]

Evaluating:  65%|██████▌   | 13/20 [00:28<00:22,  3.22s/it]

Evaluating:  70%|███████   | 14/20 [00:29<00:16,  2.67s/it]

Evaluating:  75%|███████▌  | 15/20 [00:30<00:11,  2.23s/it]

Evaluating:  85%|████████▌ | 17/20 [00:30<00:04,  1.35s/it]

Evaluating:  90%|█████████ | 18/20 [00:31<00:02,  1.19s/it]

Evaluating:  95%|█████████▌| 19/20 [00:36<00:02,  2.33s/it]

Evaluating: 100%|██████████| 20/20 [00:38<00:00,  2.06s/it]

Evaluating: 100%|██████████| 20/20 [00:38<00:00,  1.91s/it]

Out[8]:

	k	faithfulness	answer_relevancy	context_precision	context_recall
0	3	0.898	0.335	0.200	0.0
1	5	0.857	0.543	0.190	0.2
2	8	0.943	0.539	0.216	0.4

Inspect — cost of running RAGAS¶

Each metric uses an LLM judge call per sample. Track them.

In [9]:

Copied!





from cookbook import _cache
before = _cache.stats()['entries']
print(f'Cache before: {before}')
print(f'RAGAS adds ~3-4 LLM calls per sample across the four metrics.')
print(f'For 8 samples: ~30 LLM calls.')
from cookbook import _cache
before = _cache.stats()['entries']
print(f'Cache before: {before}')
print(f'RAGAS adds ~3-4 LLM calls per sample across the four metrics.')
print(f'For 8 samples: ~30 LLM calls.')

Cache before: 3504
RAGAS adds ~3-4 LLM calls per sample across the four metrics.
For 8 samples: ~30 LLM calls.

Inspect — what does the per-sample distribution look like?¶

Averages hide the spread. A pipeline with mean 0.7 and high variance is different from mean 0.7 and tight variance.

In [10]:

Copied!





print('Score interpretation guide:')
print('  faithfulness ≥ 0.75: grounded answer')
print('  answer_relevancy ≥ 0.75: addresses the question')
print('  context_precision ≥ 0.65: low junk in retrieved set')
print('  context_recall ≥ 0.55: catches required information')
print('Score interpretation guide:')
print('  faithfulness ≥ 0.75: grounded answer')
print('  answer_relevancy ≥ 0.75: addresses the question')
print('  context_precision ≥ 0.65: low junk in retrieved set')
print('  context_recall ≥ 0.55: catches required information')

Score interpretation guide:
  faithfulness ≥ 0.75: grounded answer
  answer_relevancy ≥ 0.75: addresses the question
  context_precision ≥ 0.65: low junk in retrieved set
  context_recall ≥ 0.55: catches required information

Run It¶

Run the evaluation end-to-end on the eval slice.

In [11]:

Copied!

import pandas as pd
metrics = evaluate_pipeline(answer_question, 'arxiv-mamba')
pd.DataFrame([{'metric': k, 'score': round(v, 3)} for k, v in metrics.items()])
import pandas as pd
metrics = evaluate_pipeline(answer_question, 'arxiv-mamba')
pd.DataFrame([{'metric': k, 'score': round(v, 3)} for k, v in metrics.items()])

Evaluating:   0%|          | 0/32 [00:00<?, ?it/s]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:   3%|▎         | 1/32 [00:07<03:42,  7.18s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:   9%|▉         | 3/32 [00:09<01:13,  2.54s/it]

Evaluating:  28%|██▊       | 9/32 [00:13<00:26,  1.14s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  31%|███▏      | 10/32 [00:14<00:23,  1.09s/it]

Evaluating:  38%|███▊      | 12/32 [00:14<00:15,  1.27it/s]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  41%|████      | 13/32 [00:17<00:24,  1.29s/it]

Evaluating:  44%|████▍     | 14/32 [00:18<00:19,  1.08s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  47%|████▋     | 15/32 [00:23<00:33,  1.98s/it]

Evaluating:  50%|█████     | 16/32 [00:23<00:26,  1.63s/it]

Evaluating:  56%|█████▋    | 18/32 [00:24<00:14,  1.03s/it]

Evaluating:  59%|█████▉    | 19/32 [00:24<00:13,  1.00s/it]

Evaluating:  62%|██████▎   | 20/32 [00:27<00:15,  1.31s/it]

Evaluating:  72%|███████▏  | 23/32 [00:28<00:07,  1.14it/s]

Evaluating:  75%|███████▌  | 24/32 [00:28<00:05,  1.36it/s]

Evaluating:  84%|████████▍ | 27/32 [00:30<00:03,  1.52it/s]

Evaluating:  88%|████████▊ | 28/32 [00:31<00:02,  1.38it/s]

Evaluating:  91%|█████████ | 29/32 [00:32<00:02,  1.37it/s]

Evaluating:  97%|█████████▋| 31/32 [00:35<00:00,  1.02it/s]

Evaluating: 100%|██████████| 32/32 [00:35<00:00,  1.10s/it]

Out[11]:

	metric	score
0	ragas_faithfulness	0.848
1	ragas_answer_relevancy	0.462
2	ragas_context_precision	0.128
3	ragas_context_recall	0.125

Side by Side: Vanilla Baseline vs This Technique¶

Vanilla pipeline vs the same pipeline with a reranker (loose proxy — we won't actually run rerank here to keep the cell fast). The point of the comparison is to show the evaluation harness; the recipes you've already authored have the techniques.

In [12]:

Copied!





from cookbook.baselines import vanilla_pipeline

def vanilla_fn(q):
    r = vanilla_pipeline(q, corpus='arxiv-mamba', top_k=5)
    return r.answer, r.contexts

import pandas as pd
m_vanilla = evaluate_pipeline(vanilla_fn, 'arxiv-mamba')
m_ours    = evaluate_pipeline(answer_question, 'arxiv-mamba')

pd.DataFrame([
    {'pipeline': 'vanilla', **{k.replace('ragas_', ''): round(v, 3) for k, v in m_vanilla.items()}},
    {'pipeline': 'ours',    **{k.replace('ragas_', ''): round(v, 3) for k, v in m_ours.items()}},
])
from cookbook.baselines import vanilla_pipeline

def vanilla_fn(q):
    r = vanilla_pipeline(q, corpus='arxiv-mamba', top_k=5)
    return r.answer, r.contexts

import pandas as pd
m_vanilla = evaluate_pipeline(vanilla_fn, 'arxiv-mamba')
m_ours    = evaluate_pipeline(answer_question, 'arxiv-mamba')

pd.DataFrame([
    {'pipeline': 'vanilla', **{k.replace('ragas_', ''): round(v, 3) for k, v in m_vanilla.items()}},
    {'pipeline': 'ours',    **{k.replace('ragas_', ''): round(v, 3) for k, v in m_ours.items()}},
])

Evaluating:   0%|          | 0/32 [00:00<?, ?it/s]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:   3%|▎         | 1/32 [00:05<03:00,  5.82s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:   6%|▋         | 2/32 [00:11<02:48,  5.62s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  28%|██▊       | 9/32 [00:16<00:34,  1.49s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  31%|███▏      | 10/32 [00:18<00:32,  1.46s/it]

Evaluating:  34%|███▍      | 11/32 [00:18<00:26,  1.25s/it]

Evaluating:  41%|████      | 13/32 [00:18<00:16,  1.17it/s]

Evaluating:  44%|████▍     | 14/32 [00:20<00:18,  1.04s/it]

Evaluating:  47%|████▋     | 15/32 [00:22<00:21,  1.27s/it]

Evaluating:  50%|█████     | 16/32 [00:24<00:22,  1.38s/it]

Evaluating:  56%|█████▋    | 18/32 [00:24<00:12,  1.15it/s]

Evaluating:  62%|██████▎   | 20/32 [00:29<00:17,  1.45s/it]

Evaluating:  66%|██████▌   | 21/32 [00:30<00:16,  1.47s/it]

Evaluating:  69%|██████▉   | 22/32 [00:32<00:15,  1.59s/it]

Evaluating:  78%|███████▊  | 25/32 [00:33<00:05,  1.18it/s]

Evaluating:  81%|████████▏ | 26/32 [00:33<00:05,  1.18it/s]

Evaluating:  84%|████████▍ | 27/32 [00:34<00:03,  1.40it/s]

Evaluating:  88%|████████▊ | 28/32 [00:35<00:03,  1.27it/s]

Evaluating:  91%|█████████ | 29/32 [00:37<00:03,  1.09s/it]

Evaluating:  94%|█████████▍| 30/32 [00:40<00:03,  1.77s/it]

Evaluating: 100%|██████████| 32/32 [00:40<00:00,  1.28s/it]

Evaluating:   0%|          | 0/32 [00:00<?, ?it/s]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:   3%|▎         | 1/32 [00:04<02:32,  4.92s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:   6%|▋         | 2/32 [00:10<02:47,  5.57s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  28%|██▊       | 9/32 [00:16<00:33,  1.45s/it]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  31%|███▏      | 10/32 [00:16<00:29,  1.34s/it]

Evaluating:  34%|███▍      | 11/32 [00:17<00:23,  1.14s/it]

Evaluating:  38%|███▊      | 12/32 [00:17<00:19,  1.05it/s]

LLM returned 1 generations instead of requested 3. Proceeding with 1 generations.

Evaluating:  41%|████      | 13/32 [00:21<00:32,  1.69s/it]

Evaluating:  44%|████▍     | 14/32 [00:22<00:27,  1.52s/it]

Evaluating:  47%|████▋     | 15/32 [00:25<00:33,  1.97s/it]

Evaluating:  50%|█████     | 16/32 [00:27<00:31,  1.95s/it]

Evaluating:  56%|█████▋    | 18/32 [00:29<00:20,  1.45s/it]

Evaluating:  59%|█████▉    | 19/32 [00:31<00:19,  1.52s/it]

Evaluating:  75%|███████▌  | 24/32 [00:31<00:05,  1.52it/s]

Evaluating:  81%|████████▏ | 26/32 [00:32<00:03,  1.71it/s]

Evaluating:  84%|████████▍ | 27/32 [00:35<00:04,  1.01it/s]

Evaluating:  88%|████████▊ | 28/32 [00:36<00:04,  1.02s/it]

Evaluating:  91%|█████████ | 29/32 [00:37<00:02,  1.02it/s]

Evaluating:  94%|█████████▍| 30/32 [00:38<00:02,  1.04s/it]

Evaluating: 100%|██████████| 32/32 [00:42<00:00,  1.33s/it]

Evaluating: 100%|██████████| 32/32 [00:42<00:00,  1.33s/it]

Out[12]:

	pipeline	faithfulness	answer_relevancy	context_precision	context_recall
0	vanilla	0.786	0.000	0.275	0.375
1	ours	0.869	0.464	0.128	0.125

Knobs to Turn¶

Seven knobs in priority order:

Eval-set size. 8 questions is the smoke-test floor; 80+ for confidence in rank ordering.
Judge model. Use a model at least as capable as your answer model. Mismatched judges produce noisy scores.
Metric selection. All four metrics or only some. Faithfulness and answer relevance are universally useful.
Per-corpus splits. Run eval per corpus to see where the pipeline excels and where it struggles.
Caching judge calls. RAGAS judge calls are expensive; cookbook cache helps re-runs.
Statistical tests. When comparing pipelines, run paired tests on per-sample scores, not just averages.
Per-difficulty stratification. Slice the eval set by difficulty (easy/medium/hard) and report per-bucket so improvements aren't hidden by averaging.

Evaluate on a Slice¶

Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.

In [13]:

Copied!





from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'arxiv-mamba']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'arxiv-mamba']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)

Out[13]:

	question	expected	actual	contexts_retrieved
0	What problem do state-space models aim to solv...	State-space models target the quadratic time a...	State-space models aim to solve the problem of...	5
1	Describe the selective scan mechanism introduc...	Selective scan makes the SSM parameters input-...	The passage does not describe the selective sc...	5
2	How does Mamba achieve hardware efficiency on ...	Mamba uses a parallel scan implementation with...	RecMamba achieves hardware efficiency on moder...	5
3	Which earlier model family does Mamba descend ...	Mamba builds on the structured state-space seq...	The passage does not explicitly state which ea...	5
4	Name two domains beyond text where SSM-style b...	Audio modeling and genomics have both seen suc...	Based on the provided passages, two domains be...	5

Closing Thoughts¶

Four failure modes you'll meet:

Noisy judges. Cheap judge models produce noisy scores. Use the same judge across runs to keep comparisons valid.
Small eval sets. 8 questions is enough to detect huge regressions, not enough for fine ranking. Scale up before claiming wins.
Metric over-optimisation. Tuning to RAGAS scores can produce pipelines that look great on RAGAS but fail in user studies. Always sanity-check with humans.
Cost. RAGAS is expensive at scale. Sample, don't evaluate every production query.

Compose with DeepEval (Recipe 38) for CI integration. Compose with Phoenix (Recipe 39) for trace-level diagnostics on failing samples. Compose with Lynx guardrails (Recipe 40) for production-time grounding checks.