DSPy Compiled RAG — Optimise the Prompt, Don't Engineer It¶

What problem does this solve?¶

Prompt engineering is brittle. You write a prompt that works on five queries, and the sixth produces nonsense. You add an example to the prompt; that helps the sixth but breaks the third. Iterating by hand doesn't scale, and the resulting prompts are usually one inscrutable string of instructions, few-shot examples, and edge-case patches. DSPy treats prompts as parameters to optimise, not strings to handcraft. You write a Signature (inputs/outputs) and a Module (RAG pipeline), then compile against a small training set. The optimiser searches over instruction wording, few-shot examples, and chain-of-thought structure. The compiled program routinely outperforms hand-tuned prompts and produces an artifact you can version, test, and ship like any other compiled binary.

Where it came from¶

DSPy was published by the Stanford NLP group in 2023 (Khattab et al.). The framework consolidated several earlier ideas — auto-prompting, demonstration mining, programmatic LLM use — into a single composable system. By 2026 DSPy is the canonical tool for any team that wants to ship a prompt-engineered pipeline with measurable quality bounds. The optimisers — BootstrapFewShot, MIPRO, COPRO — search the prompt space differently. The cookbook uses BootstrapFewShot for simplicity; production setups often run MIPRO for better results. The framework's appeal is that it treats prompts as software, with all the engineering machinery that implies: versioning, testing, CI integration, and reproducible compilation.

Where it fits in the RAG landscape¶

DSPy is one of several programmatic LLM tools to know:

DSPy (this recipe). Compile prompts against a training set with an optimiser.
LangChain Hub. Versioned prompt registry; no auto-optimisation.
Instructor. JSON-schema-constrained outputs; complementary to DSPy.
Promptfoo / OpenAI Evals. Evaluation-only; pair with DSPy for the compile-then-evaluate loop.

DSPy composes with everything — it's a pre-processor that produces optimised prompts, which any runtime can use. The compiled artifact is just a JSON state file plus the Python program that loads it.

When to use it (and when not to)¶

Use DSPy when you have a labelled eval set and want to systematically improve quality. Research benchmarks, production systems with feedback loops, anywhere you can measure success programmatically and want the optimiser to do the prompt-engineering work for you. Skip it when you don't have an eval set. The optimiser needs feedback to improve, and without an eval set you're tuning blindly. Skip it on one-off prototypes. The compilation overhead doesn't pay off until you're shipping a stable artifact that gets evaluated regularly.

The intuition¶

Four intuitions to carry:

Signatures are typed contracts. Each module declares inputs and outputs. The optimiser fills in the prompts that connect them, letting you focus on the data flow rather than the wording.

Optimisers search the prompt space. Bootstrap finds good few-shot examples; MIPRO co-optimises instructions and examples; COPRO refines instructions. Each one is a different search strategy over the same prompt-shape space.

Compilation is the workflow. Write the program once, compile against your eval set, ship the compiled artifact. The compile step replaces the iterative hand-tuning loop.

The metric is the optimisation target. The optimiser is only as good as the metric you give it. Spend time on the metric before spending time tuning the optimiser.

Architecture¶

flowchart TB
  T[Training set
questions+answers] --> O[Optimiser:
search prompt space]
  P[Program: signatures
+ modules] --> O
  O --> C[Compiled program
with tuned prompts]
  Q[Query] --> C
  C --> A[Answer]

References¶

💻 DSPy — Programming foundation models — Official repository.
📄 DSPy paper — The original Stanford paper.
📄 MIPRO optimiser — The MIPRO optimiser used in production DSPy setups.
📚 DSPy documentation — Tutorial and API reference.
📚 Instructor — structured outputs — JSON-schema-constrained outputs; complementary to DSPy.
📚 LangChain Hub prompt registry — Alternative for prompt versioning.

Setup¶

Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.

In [1]:

Copied!





import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())

Provider: nebius  |  Chat model: meta-llama/Llama-3.3-70B-Instruct
Tracing disabled.

Build the Pipeline, Step by Step¶

Step 1 — Build the corpus index¶

Standard Wikipedia superconductors setup.

In [2]:

Copied!





from cookbook.corpora import load_wikipedia_superconductors, load_eval_questions
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend

docs = list(load_wikipedia_superconductors())
chunks = sentence_window(docs, sentences_per_chunk=4)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('dspy', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')
from cookbook.corpora import load_wikipedia_superconductors, load_eval_questions
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend

docs = list(load_wikipedia_superconductors())
chunks = sentence_window(docs, sentences_per_chunk=4)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('dspy', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')

C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Indexed 72 chunks.

Step 2 — Configure DSPy with Nebius¶

DSPy needs an LM. We point it at the same Nebius endpoint the rest of the cookbook uses.

In [3]:

Copied!





import dspy, os

lm = dspy.LM(
    model=f'openai/{client.chat_model}',
    api_base=os.getenv(client._spec.base_url_env) if client._spec.base_url_env else None,
    api_key=os.getenv(client._spec.api_key_env),
)
dspy.configure(lm=lm)
print('DSPy configured.')
import dspy, os

lm = dspy.LM(
    model=f'openai/{client.chat_model}',
    api_base=os.getenv(client._spec.base_url_env) if client._spec.base_url_env else None,
    api_key=os.getenv(client._spec.api_key_env),
)
dspy.configure(lm=lm)
print('DSPy configured.')

17:12:02 - LiteLLM:WARNING: get_model_cost_map.py:271 - LiteLLM: Failed to fetch remote model cost map from https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json: [Errno 11001] getaddrinfo failed. Falling back to local backup.

17:12:04 - LiteLLM:WARNING: common_utils.py:979 - litellm: could not pre-load bedrock-runtime response stream shape — Bedrock event-stream decoding will be unavailable. Error: No module named 'botocore'

17:12:05 - LiteLLM:WARNING: common_utils.py:24 - litellm: could not pre-load sagemaker-runtime response stream shape — SageMaker event-stream decoding will be unavailable. Error: No module named 'botocore'

DSPy configured.

Step 3 — Define a Signature¶

Signatures declare the input/output shape. The optimiser fills in the prompt.

In [4]:

Copied!





class AnswerWithCitations(dspy.Signature):
    """Answer the question using the passages. Cite passages by their index."""
    question: str = dspy.InputField()
    passages: list[str] = dspy.InputField()
    answer: str = dspy.OutputField()

print('Signature defined.')
class AnswerWithCitations(dspy.Signature):
    """Answer the question using the passages. Cite passages by their index."""
    question: str = dspy.InputField()
    passages: list[str] = dspy.InputField()
    answer: str = dspy.OutputField()

print('Signature defined.')

Signature defined.

Step 4 — Build the RAG Module¶

A Module composes signatures. Ours is just "retrieve + answer".

In [5]:

Copied!





class CookbookRAG(dspy.Module):
    def __init__(self, k: int = 5):
        super().__init__()
        self.k = k
        self.answer = dspy.ChainOfThought(AnswerWithCitations)
    def forward(self, question: str):
        qv = client.embed([question])[0]
        hits = store.search(qv, top_k=self.k)
        return self.answer(question=question, passages=[h.text for h in hits])

raw = CookbookRAG(k=5)
print('Module built.')
print()
preview = raw(question='What is the Meissner effect?')
print(preview.answer[:300])
class CookbookRAG(dspy.Module):
    def __init__(self, k: int = 5):
        super().__init__()
        self.k = k
        self.answer = dspy.ChainOfThought(AnswerWithCitations)
    def forward(self, question: str):
        qv = client.embed([question])[0]
        hits = store.search(qv, top_k=self.k)
        return self.answer(question=question, passages=[h.text for h in hits])

raw = CookbookRAG(k=5)
print('Module built.')
print()
preview = raw(question='What is the Meissner effect?')
print(preview.answer[:300])

Module built.

The Meissner effect is the expulsion of a magnetic field from a superconductor during its transition to the superconducting state when it is cooled below the critical temperature.

Step 5 — Build a small training set¶

Pull a few labelled examples from the cookbook eval set. The optimiser uses them as ground truth.

In [6]:

Copied!





trainset = []
for row in [q for q in load_eval_questions() if q['corpus'] == 'wikipedia-superconductors'][:6]:
    trainset.append(dspy.Example(question=row['question'], answer=row['answer']).with_inputs('question'))
print(f'Training examples: {len(trainset)}')
trainset = []
for row in [q for q in load_eval_questions() if q['corpus'] == 'wikipedia-superconductors'][:6]:
    trainset.append(dspy.Example(question=row['question'], answer=row['answer']).with_inputs('question'))
print(f'Training examples: {len(trainset)}')

Training examples: 6

Step 6 — Compile with BootstrapFewShot¶

The optimiser searches for good few-shot demonstrations and a tuned instruction.

In [7]:

Copied!





def em_metric(example, pred, trace=None):
    return float(example.answer.split('.')[0].lower() in (pred.answer or '').lower())

tele = dspy.BootstrapFewShot(metric=em_metric, max_bootstrapped_demos=3, max_labeled_demos=3)
compiled = tele.compile(raw, trainset=trainset)
print('Compiled program ready.')

pred = compiled(question='What does BCS theory explain?')
print(pred.answer[:300])
def em_metric(example, pred, trace=None):
    return float(example.answer.split('.')[0].lower() in (pred.answer or '').lower())

tele = dspy.BootstrapFewShot(metric=em_metric, max_bootstrapped_demos=3, max_labeled_demos=3)
compiled = tele.compile(raw, trainset=trainset)
print('Compiled program ready.')

pred = compiled(question='What does BCS theory explain?')
print(pred.answer[:300])

  0%|          | 0/6 [00:00<?, ?it/s]

 33%|███▎      | 2/6 [00:00<00:00, 15.94it/s]

 67%|██████▋   | 4/6 [00:00<00:00, 16.04it/s]

100%|██████████| 6/6 [00:00<00:00, 15.95it/s]

100%|██████████| 6/6 [00:00<00:00, 15.96it/s]

Bootstrapped 0 full traces after 5 examples for up to 1 rounds, amounting to 6 attempts.
Compiled program ready.
BCS theory explains superconductivity as a microscopic effect caused by a condensation of pairs of electrons known as Cooper pairs, accounting for many thermodynamic and electromagnetic properties of superconductors.

Step 7 — Wrap as `answer_question`¶

Cookbook contract.

In [8]:

Copied!





def answer_question(question: str) -> tuple[str, list[str]]:
    qv = client.embed([question])[0]
    hits = store.search(qv, top_k=5)
    pred = compiled(question=question)
    return pred.answer, [h.text for h in hits]

ans, _ = answer_question('How does flux pinning enable stable levitation?')
print(ans[:400])
def answer_question(question: str) -> tuple[str, list[str]]:
    qv = client.embed([question])[0]
    hits = store.search(qv, top_k=5)
    pred = compiled(question=question)
    return pred.answer, [h.text for h in hits]

ans, _ = answer_question('How does flux pinning enable stable levitation?')
print(ans[:400])

Flux pinning enables stable levitation by preventing the movement of flux tubes within the superconductor, thereby holding it in place against the magnetic field, as described in passages 0 and 1.

Look Inside¶

Inspect — what does the compiled prompt look like?¶

DSPy stores the compiled prompt inside the program. Print it.

In [9]:

Copied!





try:
    saved = compiled.dump_state()
    import json
    print(json.dumps(saved, indent=2)[:1200])
except Exception as e:
    print(f'(dump_state failed: {e})')
try:
    saved = compiled.dump_state()
    import json
    print(json.dumps(saved, indent=2)[:1200])
except Exception as e:
    print(f'(dump_state failed: {e})')

{
  "answer.predict": {
    "traces": [],
    "train": [],
    "demos": [
      {
        "question": "Who first observed superconductivity, and in what material?",
        "answer": "Heike Kamerlingh Onnes observed it in mercury in 1911."
      },
      {
        "question": "What does BCS theory explain?",
        "answer": "It explains conventional superconductivity through electron pairing mediated by lattice vibrations, forming Cooper pairs that condense into a coherent state."
      },
      {
        "question": "What is a Cooper pair?",
        "answer": "Two electrons bound together by phonon exchange, behaving as a single boson and able to condense into a single quantum state."
      }
    ],
    "signature": {
      "instructions": "Answer the question using the passages. Cite passages by their index.",
      "fields": [
        {
          "prefix": "Question:",
          "description": "${question}"
        },
        {
          "prefix": "Passages:",
          "description": "${passages}"
        },
        {
          "prefix": "Reasoning:",
          "description": "${reasoning}"
        },
        {
          "prefix": "Answer:",
          "description": "${answer

Inspect — raw vs compiled on the same question¶

The compiled program should answer at least as well, often better.

In [10]:

Copied!





q = 'What is critical temperature?'
print('Raw answer:')
print(raw(question=q).answer[:300])
print()
print('Compiled answer:')
print(compiled(question=q).answer[:300])
q = 'What is critical temperature?'
print('Raw answer:')
print(raw(question=q).answer[:300])
print()
print('Compiled answer:')
print(compiled(question=q).answer[:300])

Raw answer:

The critical temperature is the temperature at which a material or system undergoes a significant change or transition, such as the disappearance of phase boundaries or the onset of superconductivity.

Compiled answer:
The critical temperature is the temperature below which a material becomes superconducting, characterized by zero electrical resistance and expulsion of magnetic fields.

Inspect — chain-of-thought trace¶

The ChainOfThought module exposes its reasoning. Look at it.

In [11]:

Copied!





trace = compiled(question='What is the Meissner effect?')
for attr in dir(trace):
    if attr.startswith('_'): continue
    val = getattr(trace, attr)
    if isinstance(val, str):
        print(f'{attr}: {val[:120]}')
trace = compiled(question='What is the Meissner effect?')
for attr in dir(trace):
    if attr.startswith('_'): continue
    val = getattr(trace, attr)
    if isinstance(val, str):
        print(f'{attr}: {val[:120]}')

Inspect — cost¶

Compilation costs N LLM calls per training example per optimiser step. After compilation, queries are vanilla.

In [12]:

Copied!

print('Compile-time cost: O(trainset_size * candidates * steps) LLM calls.')
print('Query-time cost: 1 LLM call per query (plus retrieval).')
print('Recommendation: cache compile-time results aggressively.')
print('Compile-time cost: O(trainset_size * candidates * steps) LLM calls.')
print('Query-time cost: 1 LLM call per query (plus retrieval).')
print('Recommendation: cache compile-time results aggressively.')

Compile-time cost: O(trainset_size * candidates * steps) LLM calls.
Query-time cost: 1 LLM call per query (plus retrieval).
Recommendation: cache compile-time results aggressively.

Run It¶

End-to-end on a representative question.

In [13]:

Copied!

ans, _ = answer_question('Explain BCS theory and its prediction of Cooper pairing.')
print('=== DSPy-compiled answer ===')
print(ans[:400])
ans, _ = answer_question('Explain BCS theory and its prediction of Cooper pairing.')
print('=== DSPy-compiled answer ===')
print(ans[:400])

=== DSPy-compiled answer ===
BCS theory explains superconductivity as a microscopic effect caused by the condensation of Cooper pairs, which are pairs of electrons bound together at low temperatures. According to the theory, these Cooper pairs move through the lattice without resistance, and their condensation is responsible for the phenomenon of superconductivity.

Side by Side: Vanilla Baseline vs This Technique¶

Vanilla vs DSPy-compiled. Same retrieval, different prompt.

In [14]:

Copied!





from cookbook.baselines import vanilla_pipeline

q = 'Explain BCS theory and its prediction of Cooper pairing.'
base = vanilla_pipeline(q, corpus='wikipedia-superconductors', top_k=5)
ours_a, _ = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'preview': base.answer[:160]},
    {'pipeline': 'dspy-compiled', 'preview': ours_a[:160]},
])
from cookbook.baselines import vanilla_pipeline

q = 'Explain BCS theory and its prediction of Cooper pairing.'
base = vanilla_pipeline(q, corpus='wikipedia-superconductors', top_k=5)
ours_a, _ = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'preview': base.answer[:160]},
    {'pipeline': 'dspy-compiled', 'preview': ours_a[:160]},
])

Out[14]:

	pipeline	preview
0	vanilla	BCS theory, or Bardeen–Cooper–Schrieffer theor...
1	dspy-compiled	BCS theory explains superconductivity as a mic...

Knobs to Turn¶

Six knobs in priority order:

Training set size. 5-10 examples are enough for BootstrapFewShot. MIPRO benefits from 30+. More is better up to a point.
Metric. Define a faithful, fast metric. The optimiser is only as good as your metric — invest time here before tuning anything else.
Optimiser. BootstrapFewShot is fast; MIPRO is slower but stronger. Try both and measure on held-out data.
Number of demos. 3-5 demonstrations is the sweet spot. Higher inflates prompt length without much quality lift.
Caching. Compilation is expensive; cache the compiled artifact and reuse across runs.
Hold-out validation set. Always evaluate the compiled artifact on data the optimiser didn't see during compilation.

Evaluate on a Slice¶

Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.

In [15]:

Copied!





from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'wikipedia-superconductors']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'wikipedia-superconductors']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)

Out[15]:

	question	expected	actual	contexts_retrieved
0	Who first observed superconductivity, and in w...	Heike Kamerlingh Onnes observed it in mercury ...	Heike Kamerlingh Onnes observed superconductiv...	5
1	What is the Meissner effect?	The complete expulsion of magnetic flux from a...	The Meissner effect is the expulsion of a magn...	5
2	Distinguish Type-I from Type-II superconductors.	Type-I has a single critical field above which...	Type-I superconductors are characterized by an...	5
3	What does BCS theory explain?	It explains conventional superconductivity thr...	BCS theory explains superconductivity as a mic...	5
4	What is a Cooper pair?	Two electrons bound together by phonon exchang...	A Cooper pair is a pair of electrons bound tog...	5

Closing Thoughts¶

Four failure modes:

Bad metric. A noisy metric leads the optimiser astray. Spend time getting the metric right.
Overfitting to small training sets. The compiled prompt may memorise examples rather than generalise. Hold out a validation set.
Compilation cost. MIPRO can spend hundreds of LLM calls per compile. Plan accordingly and cache aggressively.
Provider mismatch. A program compiled against one model may not transfer to another. Re-compile when you change the answer LM.

Compose with everything: DSPy can wrap Self-RAG modules, CRAG branches, and listwise rerankers. It is the prompt-optimisation layer that other recipes can plug into.