LangGraph Agentic RAG — Stateful Retrieval Loops¶

What problem does this solve?¶

Real questions sometimes need multiple retrieval passes. The model gets some context, realizes it needs different chunks for the next step, and retrieves again. Implementing that as a chain of function calls works for two steps but breaks down at three — the state grows, the error handling balloons, and debugging becomes archaeology. By the time you're four nodes deep, your agent looks like a bowl of spaghetti with conditional branches you can't reason about. LangGraph treats the loop as a state machine. Each retrieval, critique, and re-retrieval is a node; the transitions are edges; the state carries question, contexts, answer, and loop counter. The graph runtime handles checkpointing, retries, and conditional routing. The result: stateful agents that are debuggable as graphs, not as stack traces, and that you can visualise as Mermaid diagrams to confirm the topology matches your intent.

Where it came from¶

LangGraph shipped from LangChain in early 2024 as a stateful graph runtime for LLM agents. It built on the foundations laid by the AgentExecutor and ReAct patterns from 2022-2023, but the explicit state-graph abstraction made multi-step retrieval and tool use much easier to author and reason about. The abstraction proved sticky — by 2026 most production agentic systems run on something graph-shaped. By 2026 LangGraph and LlamaIndex Workflows are the two canonical patterns for production agentic RAG. The cookbook uses LangGraph here because of its tight integration with the LangChain ecosystem and OpenTelemetry tracing, but the LlamaIndex Workflows variant has the same shape with a slightly different API.

Where it fits in the RAG landscape¶

Three orchestration patterns to know:

Linear chain. Each step's output feeds the next. Cookbook's vanilla pipeline.
DAG (LlamaIndex Workflows). Branches and joins, no loops. Good for fan-out RAG and parallel retrieval.
State graph with cycles (this recipe). Loops, checkpointing, retries — what real agents need.

Self-RAG, CRAG, and Adaptive-RAG can all be implemented as state graphs. LangGraph is the chassis; the recipes are the patterns. The cookbook factors them as separate recipes for clarity, but in production they often live as nodes inside one bigger graph.

When to use it (and when not to)¶

Use LangGraph when your RAG needs loops, retries, or non-trivial conditional logic. Multi-hop questions, tool-using agents, anything where the next step depends on intermediate results. Skip it when a linear chain is enough. Adding a graph runtime to vanilla RAG is over-engineering. Skip it when LangChain is not your stack. LlamaIndex Workflows offers a similar abstraction with different tradeoffs.

The intuition¶

Four intuitions to internalise:

State is explicit. Every node reads and writes to a typed state dict. No hidden globals, no implicit shared state — what each node touches is visible in its signature.

Edges are conditional. A node can route to different next nodes based on the state. That's how loops and branches work; conditional routing is what makes a graph more powerful than a chain.

Caps prevent runaway. Always bound the loop count. An LLM that always asks for one more retrieval will loop forever, and you'll have an LLM bill to prove it.

Traces light up. Every node is a span in Phoenix or LangSmith. Debugging is reading the trace, and the trace is structured the way the graph is structured.

Architecture¶

flowchart TB
  START([Start]) --> RET[Retrieve node]
  RET --> CRIT[Critique node]
  CRIT -->|need more| RET
  CRIT -->|done| END([End])

References¶

📚 LangGraph documentation — The official LangGraph site.
📚 LangGraph RAG tutorial — The agentic RAG reference implementation we adapt.
📄 ReAct: Synergizing Reasoning and Acting in Language Models — The reasoning loop pattern LangGraph generalises.
📚 LlamaIndex Workflows — The DAG alternative.
📄 Self-RAG (Recipe 24) — Often implemented as a LangGraph.
📚 Anthropic agentic coding patterns — Production agentic patterns.

Setup¶

Pick a provider via the PROVIDER env var; everything below is provider-agnostic. The default is Nebius. Tracing is off by default in published notebooks so the outputs are clean — flip COOKBOOK_TRACING=phoenix to send spans to a local Phoenix UI.

In [1]:

Copied!





import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())
import os
os.environ.setdefault('PROVIDER', 'nebius')
os.environ.setdefault('COOKBOOK_TRACING', 'off')

from cookbook.providers import LLMClient
from cookbook.tracing import init_tracing

client = LLMClient()
print(f'Provider: {client.provider}  |  Chat model: {client.chat_model}')
print(init_tracing())

Provider: nebius  |  Chat model: meta-llama/Llama-3.3-70B-Instruct
Tracing disabled.

Build the Pipeline, Step by Step¶

Step 1 — Build the index¶

Standard Rust book setup.

In [2]:

Copied!





from cookbook.corpora import load_rust_book
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend

docs = list(load_rust_book())
chunks = sentence_window(docs, sentences_per_chunk=4)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('lg', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')
from cookbook.corpora import load_rust_book
from cookbook.chunkers import sentence_window
from cookbook.stores import QdrantBackend

docs = list(load_rust_book())
chunks = sentence_window(docs, sentences_per_chunk=4)
vectors = client.embed([c.text for c in chunks])
store = QdrantBackend('lg', dim=len(vectors[0]))
store.add([c.text for c in chunks], vectors, ids=[c.chunk_id for c in chunks])
print(f'Indexed {len(chunks)} chunks.')

C:\Users\faree\Desktop\rag\rag-cookbook-2026\.venv\Lib\site-packages\tqdm\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Indexed 173 chunks.

Step 2 — Define the state¶

Typed dict. Each field is read and written by various nodes. LangGraph routes based on these values.

In [3]:

Copied!





from typing import TypedDict

class State(TypedDict):
    question: str
    query: str
    contexts: list[str]
    answer: str
    loops: int
print('State schema defined.')
from typing import TypedDict

class State(TypedDict):
    question: str
    query: str
    contexts: list[str]
    answer: str
    loops: int
print('State schema defined.')

State schema defined.

Step 3 — Define the retrieve and critique nodes¶

Each node takes the state and returns a dict of updates. The runtime merges them into the state.

In [4]:

Copied!





def retrieve(s: State) -> dict:
    qv = client.embed([s['query']])[0]
    hits = store.search(qv, top_k=5)
    new_contexts = s.get('contexts', []) + [h.text for h in hits]
    return {'contexts': new_contexts}

def critique(s: State) -> dict:
    raw = client.chat(
        'Given the question and retrieved passages, reply with EITHER:\n'
        '  ANSWER: <final answer>\n  NEED: <a more specific follow-up query>\n'
        f'Question: {s["question"]}\nPassages:\n'
        + '\n\n'.join(s['contexts'][-10:])
    )
    if raw.strip().upper().startswith('ANSWER'):
        return {'answer': raw.split(':', 1)[1].strip()}
    return {'query': raw.split(':', 1)[1].strip(), 'loops': s.get('loops', 0) + 1}

print('Nodes defined.')
def retrieve(s: State) -> dict:
    qv = client.embed([s['query']])[0]
    hits = store.search(qv, top_k=5)
    new_contexts = s.get('contexts', []) + [h.text for h in hits]
    return {'contexts': new_contexts}

def critique(s: State) -> dict:
    raw = client.chat(
        'Given the question and retrieved passages, reply with EITHER:\n'
        '  ANSWER: <final answer>\n  NEED: <a more specific follow-up query>\n'
        f'Question: {s["question"]}\nPassages:\n'
        + '\n\n'.join(s['contexts'][-10:])
    )
    if raw.strip().upper().startswith('ANSWER'):
        return {'answer': raw.split(':', 1)[1].strip()}
    return {'query': raw.split(':', 1)[1].strip(), 'loops': s.get('loops', 0) + 1}

print('Nodes defined.')

Nodes defined.

Step 4 — Build the graph¶

Start at retrieve, route through critique, loop back if needed, exit when we have an answer or hit the loop cap.

In [5]:

Copied!





from langgraph.graph import StateGraph, END

def decide(s: State) -> str:
    if s.get('answer'):
        return END
    if s.get('loops', 0) >= 3:
        return END
    return 'retrieve'

g = StateGraph(State)
g.add_node('retrieve', retrieve)
g.add_node('critique', critique)
g.set_entry_point('retrieve')
g.add_edge('retrieve', 'critique')
g.add_conditional_edges('critique', decide, {'retrieve': 'retrieve', END: END})
app = g.compile()
print('Graph compiled.')
from langgraph.graph import StateGraph, END

def decide(s: State) -> str:
    if s.get('answer'):
        return END
    if s.get('loops', 0) >= 3:
        return END
    return 'retrieve'

g = StateGraph(State)
g.add_node('retrieve', retrieve)
g.add_node('critique', critique)
g.set_entry_point('retrieve')
g.add_edge('retrieve', 'critique')
g.add_conditional_edges('critique', decide, {'retrieve': 'retrieve', END: END})
app = g.compile()
print('Graph compiled.')

Graph compiled.

Step 5 — Run the graph¶

Invoke with initial state. The runtime walks the nodes until END.

In [6]:

Copied!





result = app.invoke({
    'question': 'When should I prefer Arc over Rc, and what about RefCell?',
    'query': 'Rc Arc RefCell trade-offs',
    'contexts': [],
    'loops': 0,
})
print(f'Final answer:\n{result.get("answer", "(no answer)")[:400]}')
print()
print(f'Total loops: {result.get("loops", 0)}')
print(f'Contexts collected: {len(result.get("contexts", []))}')
result = app.invoke({
    'question': 'When should I prefer Arc over Rc, and what about RefCell?',
    'query': 'Rc Arc RefCell trade-offs',
    'contexts': [],
    'loops': 0,
})
print(f'Final answer:\n{result.get("answer", "(no answer)")[:400]}')
print()
print(f'Total loops: {result.get("loops", 0)}')
print(f'Contexts collected: {len(result.get("contexts", []))}')

17:42:05 - LiteLLM:WARNING: common_utils.py:979 - litellm: could not pre-load bedrock-runtime response stream shape — Bedrock event-stream decoding will be unavailable. Error: No module named 'botocore'

17:42:05 - LiteLLM:WARNING: common_utils.py:24 - litellm: could not pre-load sagemaker-runtime response stream shape — SageMaker event-stream decoding will be unavailable. Error: No module named 'botocore'

Final answer:
(no answer)

Total loops: 3
Contexts collected: 15

Step 6 — Wrap as `answer_question`¶

Cookbook contract.

In [7]:

Copied!





def answer_question(question: str) -> tuple[str, list[str]]:
    result = app.invoke({
        'question': question,
        'query': question,
        'contexts': [],
        'loops': 0,
    })
    return result.get('answer', '(no answer)'), result.get('contexts', [])[-5:]

ans, _ = answer_question('When should I prefer Arc over Rc, and what about RefCell?')
print(ans[:400])
def answer_question(question: str) -> tuple[str, list[str]]:
    result = app.invoke({
        'question': question,
        'query': question,
        'contexts': [],
        'loops': 0,
    })
    return result.get('answer', '(no answer)'), result.get('contexts', [])[-5:]

ans, _ = answer_question('When should I prefer Arc over Rc, and what about RefCell?')
print(ans[:400])

(no answer)

Look Inside¶

Inspect — visualise the graph¶

LangGraph can render the compiled graph as Mermaid for easy inspection.

In [8]:

Copied!





try:
    mermaid = app.get_graph().draw_mermaid()
    print(mermaid)
except Exception as e:
    print(f'(Mermaid render failed: {e})')
try:
    mermaid = app.get_graph().draw_mermaid()
    print(mermaid)
except Exception as e:
    print(f'(Mermaid render failed: {e})')

---
config:
  flowchart:
    curve: linear
---
graph TD;
	__start__([<p>__start__</p>]):::first
	retrieve(retrieve)
	critique(critique)
	__end__([<p>__end__</p>]):::last
	__start__ --> retrieve;
	critique -.-> __end__;
	critique -.-> retrieve;
	retrieve --> critique;
	classDef default fill:#f2f0ff,line-height:1.2
	classDef first fill-opacity:0
	classDef last fill:#bfb6fc

Inspect — trace the loop count¶

Run several questions, count how many loops each takes. Simple questions resolve in one loop; harder ones bounce.

In [9]:

Copied!





import pandas as pd
rows = []
for q in [
    'What is Rc?',                                    # easy
    'When do I use Arc instead of Rc?',               # easy
    'How do Arc and Mutex compose for shared state?', # medium
    'Walk through how lifetimes affect a function that returns its longest argument.',  # hard
]:
    result = app.invoke({'question': q, 'query': q, 'contexts': [], 'loops': 0})
    rows.append({'q': q[:55], 'loops': result.get('loops', 0), 'has_answer': bool(result.get('answer'))})
pd.DataFrame(rows)
import pandas as pd
rows = []
for q in [
    'What is Rc?',                                    # easy
    'When do I use Arc instead of Rc?',               # easy
    'How do Arc and Mutex compose for shared state?', # medium
    'Walk through how lifetimes affect a function that returns its longest argument.',  # hard
]:
    result = app.invoke({'question': q, 'query': q, 'contexts': [], 'loops': 0})
    rows.append({'q': q[:55], 'loops': result.get('loops', 0), 'has_answer': bool(result.get('answer'))})
pd.DataFrame(rows)

Out[9]:

	q	loops	has_answer
0	What is Rc?	0	True
1	When do I use Arc instead of Rc?	3	False
2	How do Arc and Mutex compose for shared state?	3	False
3	Walk through how lifetimes affect a function t...	3	False

Inspect — what does the critique propose at each loop?¶

Look at the rewrite proposed at each iteration. Good rewrites narrow the search; bad ones repeat or drift.

In [10]:

Copied!





# Step through manually with prints
s: State = {'question': 'How do Arc and Mutex compose?', 'query': 'Arc Mutex', 'contexts': [], 'loops': 0}
for step in range(3):
    s.update(retrieve(s))
    print(f'Loop {step+1} retrieved {len(s["contexts"])} contexts so far.')
    update = critique(s)
    if 'answer' in update:
        print(f'  Critique decided to answer.')
        s.update(update)
        break
    print(f'  Next query: {update["query"]}')
    s.update(update)
# Step through manually with prints
s: State = {'question': 'How do Arc and Mutex compose?', 'query': 'Arc Mutex', 'contexts': [], 'loops': 0}
for step in range(3):
    s.update(retrieve(s))
    print(f'Loop {step+1} retrieved {len(s["contexts"])} contexts so far.')
    update = critique(s)
    if 'answer' in update:
        print(f'  Critique decided to answer.')
        s.update(update)
        break
    print(f'  Next query: {update["query"]}')
    s.update(update)

Loop 1 retrieved 5 contexts so far.

  Next query: More specific information about Arc and Mutex, as the provided passages do not mention their composition.

Loop 2 retrieved 10 contexts so far.

  Next query: More specific information about Arc and Mutex, as the provided passages do not explicitly discuss how Arc and Mutex compose.

Loop 3 retrieved 15 contexts so far.

  Next query: More specific information about Arc and Mutex, as the provided passages do not explicitly discuss how Arc and Mutex compose.

Inspect — cost¶

Each loop is one retrieval + one critique LLM call. We measure.

In [11]:

Copied!





from cookbook import _cache
before = _cache.stats()['entries']
_ = answer_question('How does the borrow checker reason about overlapping references?')
after = _cache.stats()['entries']
print(f'New cache entries: {after - before}')
print('Each loop: 1 retrieval embed + 1 critique LLM call.')
from cookbook import _cache
before = _cache.stats()['entries']
_ = answer_question('How does the borrow checker reason about overlapping references?')
after = _cache.stats()['entries']
print(f'New cache entries: {after - before}')
print('Each loop: 1 retrieval embed + 1 critique LLM call.')

New cache entries: 5
Each loop: 1 retrieval embed + 1 critique LLM call.

Run It¶

End-to-end on a multi-hop question.

In [12]:

Copied!

ans, _ = answer_question('Walk through how the borrow checker reasons about overlapping references in a function that returns the longer of two arguments.')
print('=== LangGraph answer ===')
print(ans)
ans, _ = answer_question('Walk through how the borrow checker reasons about overlapping references in a function that returns the longer of two arguments.')
print('=== LangGraph answer ===')
print(ans)

=== LangGraph answer ===
(no answer)

Side by Side: Vanilla Baseline vs This Technique¶

Vanilla (single retrieval) vs LangGraph (looping). On multi-hop questions the loop adds value; on single-hop, the loop is wasted work but doesn't hurt quality.

In [13]:

Copied!





from cookbook.baselines import vanilla_pipeline

q = 'Walk through how the borrow checker reasons about overlapping references.'
base = vanilla_pipeline(q, corpus='rust-book', top_k=5)
ours_a, _ = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'preview': base.answer[:160]},
    {'pipeline': 'langgraph', 'preview': ours_a[:160]},
])
from cookbook.baselines import vanilla_pipeline

q = 'Walk through how the borrow checker reasons about overlapping references.'
base = vanilla_pipeline(q, corpus='rust-book', top_k=5)
ours_a, _ = answer_question(q)

import pandas as pd
pd.DataFrame([
    {'pipeline': 'vanilla', 'preview': base.answer[:160]},
    {'pipeline': 'langgraph', 'preview': ours_a[:160]},
])

Out[13]:

	pipeline	preview
0	vanilla	The passages do not contain the answer. They d...
1	langgraph	(no answer)

Knobs to Turn¶

Six knobs in priority order:

Loop cap. Default 3. Higher allows deeper multi-hop but invites runaway. Cap aggressively in production — a hung agent is worse than a partial answer.
Critique prompt. Specific failure-mode prompts work better than generic "is this answer good". Phrase it as "what evidence is still missing?".
Retrieval k per loop. We use 5. Lower keeps the context tight; higher gathers more evidence per loop.
State persistence. LangGraph supports checkpointing — useful for long-running agents and resumable conversations across page refreshes.
Tracing. Always enable Phoenix or LangSmith. Debugging an agent without traces is essentially impossible.
Node-level retries. Wrap each node in a retry policy so transient model failures don't kill the whole loop.

Evaluate on a Slice¶

Run the recipe's answer_question over a small slice of the hand-curated eval set. Full RAGAS metrics are exercised in recipes/09-evaluation-and-production/ragas-triad-eval.ipynb; here we just print a quick spot-check table so you can eyeball whether the technique is on track.

In [14]:

Copied!





from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'rust-book']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)
from cookbook.corpora import load_eval_questions
from cookbook.eval import EvalSample

qs = load_eval_questions()
qs = [q for q in qs if q['corpus'] == 'rust-book']
samples = []
for row in qs[:5]:
    answer, contexts = answer_question(row['question'])
    samples.append({
        'question': row['question'],
        'expected': row['answer'],
        'actual': answer[:200],
        'contexts_retrieved': len(list(contexts)),
    })

import pandas as pd
pd.DataFrame(samples)

Out[14]:

	question	expected	actual	contexts_retrieved
0	What is ownership in Rust?	A set of rules governing how memory is managed...	Ownership in Rust is a feature that enables th...	5
1	What does the borrow checker do?	It statically enforces that references obey th...	The borrow checker ensures that references to ...	5
2	What is the difference between String and &str?	`String` is an owned, heap-allocated, growable...	(no answer)	5
3	Describe how match is exhaustive in Rust.	The compiler requires `match` arms to cover ev...	In Rust, a `match` is exhaustive because the c...	5
4	What is a trait?	A trait is a named set of methods that types c...	A trait is a way to define behavior in a gener...	5

Closing Thoughts¶

Four failure modes you'll meet in production:

Runaway loops. Without the cap the agent loops forever. Always cap.
Drifting critique. The critique proposes worse follow-up questions over time. Tighten the critique prompt or reset the state on each loop.
State bloat. Contexts accumulate; the prompt eventually overflows. Truncate or summarise older contexts.
Cyclic loops with no progress. The critique keeps asking for the same retrieval. Detect by hashing the query each loop and break on repeat.

Compose with Self-RAG (Recipe 24) inside critique, CRAG (Recipe 25) for fallback, and adaptive routing (Recipe 26) to decide whether to invoke the graph at all.