DeepEval — Metrics in Your CI Pipeline¶

What problem does this solve?¶

RAGAS gives you metrics on demand. CI gives you metrics on every commit. Without CI integration, RAG quality regressions ship to production silently. DeepEval treats LLM evaluations as pytest tests: write a metric threshold, the CI build fails if a PR drops below it. The setup takes a few hours; the savings show up the first time a refactor would have shipped a regression. The CI gate is what makes RAG evolution sustainable. Without it, every team eventually hits the same pattern — quality drifts down quietly, users complain, someone runs a manual eval, the team scrambles. The gate prevents the silent drift by forcing every PR to defend its quality bar before merging.

Where it came from¶

DeepEval was released by Confident AI in 2024 as a pytest-first LLM evaluation framework. Its design choice — LLMTestCase objects, metric classes with measure() and is_successful(), pytest decorators — made it the natural fit for CI pipelines. By 2026 DeepEval is the default for teams that want quality gates without writing custom pytest fixtures, and the framework has settled into a mature shape with stable metric definitions. The pytest-first design is the reason it sticks. Engineers already know pytest; adding LLM tests doesn't introduce a new framework. CI pipelines need no special integration; DeepEval tests run alongside unit tests under the same pytest invocation, get the same pass/fail signals, and ship through the same release-gate machinery.

Where it fits in the RAG landscape¶

Where DeepEval fits among RAG evaluation tools:

RAGAS (Recipe 37). Notebook-first, ad-hoc evaluation. Best for exploration and one-off analysis.
DeepEval (this recipe). Pytest-first, CI evaluation with hard pass/fail thresholds.
Phoenix (Recipe 39). Trace-first, debugging. Best when you want to inspect individual failures.
Langfuse. Production observability with cost tracking.

All four overlap on metrics; they differ on workflow integration. Pick by where in your stack the evaluation lives. DeepEval is the only one designed to gate releases via pytest, so it's the natural choice for CI-driven teams.

When to use it (and when not to)¶

Use DeepEval to gate releases. Any team shipping RAG changes through PR should have a DeepEval test suite that runs on every PR. Skip it for exploratory work. RAGAS notebook usage is faster for iteration. Skip it when CI is slow. DeepEval suites that take 30 minutes will get bypassed; design for fast feedback.

The intuition¶

Four intuitions:

Tests are assertions on metrics. Each test creates an LLMTestCase, calls a metric's measure(), asserts that is_successful().

Thresholds are the contract. Each metric has a threshold (default 0.7). Below threshold = test fails = PR blocked.

Cache the judge calls. Without caching, DeepEval re-runs every judge on every CI build. With caching, only changed pipelines re-evaluate.

Subset the eval set. Run a fast subset on every PR, the full set nightly.

Architecture¶

flowchart LR
  PR[PR] --> CI[CI build]
  CI --> T[Run pytest
with DeepEval suite]
  T --> M[Each test:
measure metric]
  M --> P{All pass?}
  P -->|yes| MERGE[Allow merge]
  P -->|no| BLOCK[Block merge]

References¶

📚 DeepEval documentation — Official docs.
💻 confident-ai/deepeval repository — Source.
📚 DeepEval metrics overview — The metrics catalogue.
📚 pytest fixtures for LLM evaluation — The pytest mechanism DeepEval builds on.
📚 RAGAS (Recipe 37) — The notebook-first alternative.
📚 GitHub Actions for CI — Where the DeepEval suite typically runs.

Setup¶

Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.

In [1]:

Copied!





import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())

Provider: nebius  |  Chat model: meta-llama/Llama-3.3-70B-Instruct
Tracing disabled.

Build the Pipeline, Step by Step¶

Step 1 — Build a small pipeline¶

Standard Rust book RAG. We'll write tests against it.

In [2]:

Copied!





from cookbook.corpora import load_rust_book
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend

docs = list(load_rust_book())
chunks = sentence_window(docs, sentences_per_chunk=4)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('deepeval', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])

def answer_question(question: str) -> tuple[str, list[str]]:
    qv = client.embed([question])[0]
    hits = store.search(qv, top_k=5)
    ctx = [h.text for h in hits]
    return client.chat('Use these.\n' + '\n\n'.join(ctx) + f'\nQ: {question}\nA:'), ctx

print('Pipeline ready.')
from cookbook.corpora import load_rust_book
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend

docs = list(load_rust_book())
chunks = sentence_window(docs, sentences_per_chunk=4)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('deepeval', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])

def answer_question(question: str) -> tuple[str, list[str]]:
    qv = client.embed([question])[0]
    hits = store.search(qv, top_k=5)
    ctx = [h.text for h in hits]
    return client.chat('Use these.\n' + '\n\n'.join(ctx) + f'\nQ: {question}\nA:'), ctx

print('Pipeline ready.')

C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Pipeline ready.

Step 2 — Build a single DeepEval test case¶

An LLMTestCase carries everything DeepEval needs: input, output, expected output, retrieved context.

In [3]:

Copied!





import os
os.environ.setdefault('OPENAI_API_KEY', os.environ.get(client._spec.api_key_env, ''))
os.environ.setdefault('OPENAI_BASE_URL', os.environ.get(client._spec.base_url_env, '') or '')

from deepeval.test_case import LLMTestCase
from cookbook.eval import _build_deepeval_model
judge = _build_deepeval_model()

q = 'What is ownership in Rust?'
ans, ctx = answer_question(q)
tc = LLMTestCase(
    input=q,
    actual_output=ans,
    expected_output='Ownership is a set of rules that govern memory management in Rust.',
    retrieval_context=ctx,
)
print(f'Test case input: {tc.input[:60]}')
print(f'Test case actual_output: {tc.actual_output[:120]}')
import os
os.environ.setdefault('OPENAI_API_KEY', os.environ.get(client._spec.api_key_env, ''))
os.environ.setdefault('OPENAI_BASE_URL', os.environ.get(client._spec.base_url_env, '') or '')

from deepeval.test_case import LLMTestCase
from cookbook.eval import _build_deepeval_model
judge = _build_deepeval_model()

q = 'What is ownership in Rust?'
ans, ctx = answer_question(q)
tc = LLMTestCase(
    input=q,
    actual_output=ans,
    expected_output='Ownership is a set of rules that govern memory management in Rust.',
    retrieval_context=ctx,
)
print(f'Test case input: {tc.input[:60]}')
print(f'Test case actual_output: {tc.actual_output[:120]}')

Test case input: What is ownership in Rust?
Test case actual_output: Ownership in Rust refers to a unique feature of the language that enables it to guarantee memory safety without the need

Step 3 — Run a single metric¶

FaithfulnessMetric reads the test case and produces a score. is_successful() checks against the threshold.

In [4]:

Copied!





from deepeval.metrics import FaithfulnessMetric

m = FaithfulnessMetric(threshold=0.7, model=judge)
m.measure(tc)
print(f'Faithfulness score: {m.score:.3f}')
print(f'Is successful: {m.is_successful()}')
from deepeval.metrics import FaithfulnessMetric

m = FaithfulnessMetric(threshold=0.7, model=judge)
m.measure(tc)
print(f'Faithfulness score: {m.score:.3f}')
print(f'Is successful: {m.is_successful()}')

C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\rich\live.py:260: UserWarning: install 
"ipywidgets" for Jupyter support
  warnings.warn('install "ipywidgets" for Jupyter support')

Faithfulness score: 1.000
Is successful: True

Step 4 — Write the pytest file¶

DeepEval tests are normal pytest. The file goes in tests/test_metrics.py and is picked up by pytest.

In [5]:

Copied!





TEST = '''
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_baseline_faithfulness():
    tc = LLMTestCase(
        input='What is ownership in Rust?',
        actual_output='Ownership is a set of rules that govern how Rust manages memory.',
        expected_output='Ownership rules govern memory management.',
        retrieval_context=['Ownership is Rust's most unique feature; it has deep implications for memory safety.'],
    )
    m = FaithfulnessMetric(threshold=0.7)
    m.measure(tc)
    assert m.is_successful(), f'faithfulness={m.score}'

def test_baseline_relevancy():
    tc = LLMTestCase(
        input='What is ownership in Rust?',
        actual_output='Ownership is a set of rules that govern how Rust manages memory.',
        expected_output='Ownership rules govern memory management.',
        retrieval_context=['Ownership is Rust's most unique feature.'],
    )
    m = AnswerRelevancyMetric(threshold=0.7)
    m.measure(tc)
    assert m.is_successful(), f'answer_relevancy={m.score}'
'''
from pathlib import Path
p = Path('/tmp/test_metrics.py')
p.write_text(TEST, encoding='utf-8')
print(f'Wrote {p}')
print()
print('Run with: pytest /tmp/test_metrics.py')
TEST = '''
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_baseline_faithfulness():
    tc = LLMTestCase(
        input='What is ownership in Rust?',
        actual_output='Ownership is a set of rules that govern how Rust manages memory.',
        expected_output='Ownership rules govern memory management.',
        retrieval_context=['Ownership is Rust's most unique feature; it has deep implications for memory safety.'],
    )
    m = FaithfulnessMetric(threshold=0.7)
    m.measure(tc)
    assert m.is_successful(), f'faithfulness={m.score}'

def test_baseline_relevancy():
    tc = LLMTestCase(
        input='What is ownership in Rust?',
        actual_output='Ownership is a set of rules that govern how Rust manages memory.',
        expected_output='Ownership rules govern memory management.',
        retrieval_context=['Ownership is Rust's most unique feature.'],
    )
    m = AnswerRelevancyMetric(threshold=0.7)
    m.measure(tc)
    assert m.is_successful(), f'answer_relevancy={m.score}'
'''
from pathlib import Path
p = Path('/tmp/test_metrics.py')
p.write_text(TEST, encoding='utf-8')
print(f'Wrote {p}')
print()
print('Run with: pytest /tmp/test_metrics.py')

Wrote \tmp\test_metrics.py

Run with: pytest /tmp/test_metrics.py

Step 5 — Wrap as `answer_question`¶

Cookbook contract.

In [6]:

Copied!

# `answer_question` was defined earlier; nothing new to wire here.
ans, _ = answer_question('What is borrowing?')
print(ans[:200])
# `answer_question` was defined earlier; nothing new to wire here.
ans, _ = answer_question('What is borrowing?')
print(ans[:200])

Borrowing is a related feature to ownership in Rust, which will be discussed in this chapter along with slices and how Rust lays out data in memory.

Look Inside¶

Inspect — score multiple metrics on one test case¶

Each metric is an independent assertion; CI builds usually run several.

In [7]:

Copied!





from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric, ContextualPrecisionMetric

for M in (FaithfulnessMetric, AnswerRelevancyMetric, ContextualPrecisionMetric):
    m = M(threshold=0.7, model=judge)
    m.measure(tc)
    print(f'  {M.__name__:30s}  score={m.score:.3f}  pass={m.is_successful()}')
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric, ContextualPrecisionMetric

for M in (FaithfulnessMetric, AnswerRelevancyMetric, ContextualPrecisionMetric):
    m = M(threshold=0.7, model=judge)
    m.measure(tc)
    print(f'  {M.__name__:30s}  score={m.score:.3f}  pass={m.is_successful()}')

  FaithfulnessMetric              score=1.000  pass=True

  AnswerRelevancyMetric           score=1.000  pass=True

  ContextualPrecisionMetric       score=0.750  pass=True

Inspect — design a CI subset and a nightly suite¶

The CI suite should run in under 5 minutes. Subset aggressively.

In [8]:

Copied!





print('Recommended CI design:')
print('  PR suite: 5-10 representative tests, <5 min runtime.')
print('  Nightly:  full 80+ question eval set with all metrics.')
print('  Quarterly: human eval on a labelled subset.')
print('Recommended CI design:')
print('  PR suite: 5-10 representative tests, <5 min runtime.')
print('  Nightly:  full 80+ question eval set with all metrics.')
print('  Quarterly: human eval on a labelled subset.')

Recommended CI design:
  PR suite: 5-10 representative tests, <5 min runtime.
  Nightly:  full 80+ question eval set with all metrics.
  Quarterly: human eval on a labelled subset.

Inspect — how to handle flakiness¶

LLM judges are noisy. Plan for retries.

In [9]:

Copied!





print('Flakiness mitigation:')
print('  pytest-retry for transient flakes.')
print('  Lower thresholds on CI than on quarterly review.')
print('  Compute confidence intervals on the eval set; reject only outside CI.')
print('Flakiness mitigation:')
print('  pytest-retry for transient flakes.')
print('  Lower thresholds on CI than on quarterly review.')
print('  Compute confidence intervals on the eval set; reject only outside CI.')

Flakiness mitigation:
  pytest-retry for transient flakes.
  Lower thresholds on CI than on quarterly review.
  Compute confidence intervals on the eval set; reject only outside CI.

Inspect — cost¶

DeepEval costs one judge call per test per metric. At 10 tests × 3 metrics × $0.001 per judge call = $0.03 per CI run.

In [10]:

Copied!





n_tests = 10
n_metrics = 3
cost_per_judge = 0.001  # rough
total_cost = n_tests * n_metrics * cost_per_judge
print(f'Estimated cost per CI run: ${total_cost:.2f}')
n_tests = 10
n_metrics = 3
cost_per_judge = 0.001  # rough
total_cost = n_tests * n_metrics * cost_per_judge
print(f'Estimated cost per CI run: ${total_cost:.2f}')

Estimated cost per CI run: $0.03

Run It¶

Run a small DeepEval suite against the pipeline.

In [11]:

Copied!





from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

from cookbook.corpora import load_eval_questions
results = []
for row in [q for q in load_eval_questions() if q['corpus'] == 'rust-book'][:3]:
    actual, ctx = answer_question(row['question'])
    tc = LLMTestCase(
        input=row['question'], actual_output=actual,
        expected_output=row['answer'], retrieval_context=ctx,
    )
    fm = FaithfulnessMetric(threshold=0.7, model=judge); fm.measure(tc)
    am = AnswerRelevancyMetric(threshold=0.7, model=judge); am.measure(tc)
    results.append({
        'q': row['question'][:50],
        'faithfulness': round(fm.score, 2),
        'relevancy': round(am.score, 2),
    })
import pandas as pd
pd.DataFrame(results)
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

from cookbook.corpora import load_eval_questions
results = []
for row in [q for q in load_eval_questions() if q['corpus'] == 'rust-book'][:3]:
    actual, ctx = answer_question(row['question'])
    tc = LLMTestCase(
        input=row['question'], actual_output=actual,
        expected_output=row['answer'], retrieval_context=ctx,
    )
    fm = FaithfulnessMetric(threshold=0.7, model=judge); fm.measure(tc)
    am = AnswerRelevancyMetric(threshold=0.7, model=judge); am.measure(tc)
    results.append({
        'q': row['question'][:50],
        'faithfulness': round(fm.score, 2),
        'relevancy': round(am.score, 2),
    })
import pandas as pd
pd.DataFrame(results)

Out[11]:

	q	faithfulness	relevancy
0	What is ownership in Rust?	1.00	1.00
1	What does the borrow checker do?	1.00	0.75
2	What is the difference between String and &str?	0.86	0.90

Side by Side: Vanilla Baseline vs This Technique¶

Vanilla vs vanilla — DeepEval is a measurement tool, not a pipeline; the comparison just shows the same pipeline can be measured.

In [12]:

Copied!





import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'notes': 'Measured with DeepEval Faithfulness / AnswerRelevancy metrics.'},
])
import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'notes': 'Measured with DeepEval Faithfulness / AnswerRelevancy metrics.'},
])

Out[12]:

	pipeline	notes
0	vanilla	Measured with DeepEval Faithfulness / AnswerRe...

Knobs to Turn¶

Six knobs in priority order:

Threshold per metric. Default 0.7. Higher fails more PRs; lower lets through regressions. Calibrate on a labelled set.
Test selection. Pick representative tests for CI; full set for nightly. Aim for under 5 minute CI feedback.
Judge model. Mid-tier is fine. Bigger judges add cost without quality lift.
Caching. Cache judge calls so unchanged code doesn't re-evaluate. Keys on (model, input, output).
Retry on flakes. pytest-retry handles transient judge failures and provider rate limits.
Parametrise eval data. Use pytest's parametrize to run the same metric across many test cases without code duplication.

Evaluate on a Slice¶

Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.

In [13]:

Copied!





from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'rust-book']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'rust-book']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)

Out[13]:

	question	expected	actual	contexts_retrieved
0	What is ownership in Rust?	A set of rules governing how memory is managed...	Ownership in Rust refers to a unique feature o...	5
1	What does the borrow checker do?	It statically enforces that references obey th...	The borrow checker is not explicitly mentioned...	5
2	What is the difference between String and &str?	`String` is an owned, heap-allocated, growable...	The text does not explicitly state the differe...	5
3	Describe how match is exhaustive in Rust.	The compiler requires `match` arms to cover ev...	In Rust, `match` is exhaustive, meaning that t...	5
4	What is a trait?	A trait is a named set of methods that types c...	A trait in Rust is a way to define a set of me...	5

Closing Thoughts¶

Four failure modes you'll meet:

Flakes. LLM judges aren't deterministic. Retries handle most; the rest needs threshold tolerance.
Tests too slow. Engineers will bypass slow CI. Subset aggressively to keep feedback under 5 minutes.
Over-fitting to tests. Tuning to the CI suite produces pipelines that pass CI but fail in production. Diversify the test set.
Schema drift. When the cookbook eval JSONL changes, DeepEval tests against old schemas break silently. Pin schemas.

Compose with RAGAS (Recipe 37) for ad-hoc analysis, Phoenix (Recipe 39) for trace inspection on failing tests, and Lynx guardrails (Recipe 40) for production-time enforcement.