Skip to content

cookbook.corpora

cookbook.corpora

Loaders for the four cookbook corpora.

Every recipe pulls its documents through one of the loaders here. The actual files live under corpus/ and are downloaded reproducibly by scripts/fetch_corpus.py. The loaders return iterables of Document dicts with the schema below so every vector-store backend, chunker, and evaluator speaks the same language.

Schema

{ "doc_id": str, # unique within the corpus "source": str, # human-readable origin "text": str, # plain text content "metadata": dict, # arbitrary; commonly {section, page, url} }

Document dataclass

One unit of corpus material — page, section, or whole file.

Source code in cookbook/corpora.py 
28
29
30
31
32
33
34
35
@dataclass(frozen=True)
class Document:
    """One unit of corpus material — page, section, or whole file."""

    doc_id: str
    source: str
    text: str
    metadata: dict

load_arxiv_mamba()

A recent arXiv survey of state-space sequence models.

Returns one Document per page. Used by recipes that want a dense, technical long document to chunk and reason over.

Source code in cookbook/corpora.py 
41
42
43
44
45
46
47
48
def load_arxiv_mamba() -> Iterator[Document]:
    """A recent arXiv survey of state-space sequence models.

    Returns one Document per page. Used by recipes that want a dense, technical
    long document to chunk and reason over.
    """
    pdf = CORPUS_DIR / "arxiv-2403-mamba.pdf"
    yield from _yield_pdf_pages(pdf, source="arxiv:2403-mamba-survey")

load_wikipedia_superconductors()

About 50 markdown pages on superconductivity from Wikipedia.

One Document per markdown file. Good for graph-RAG and multi-hop queries.

Source code in cookbook/corpora.py 
51
52
53
54
55
56
57
58
59
60
61
62
63
def load_wikipedia_superconductors() -> Iterator[Document]:
    """About 50 markdown pages on superconductivity from Wikipedia.

    One Document per markdown file. Good for graph-RAG and multi-hop queries.
    """
    folder = CORPUS_DIR / "wikipedia-superconductors"
    for md in sorted(folder.glob("*.md")):
        yield Document(
            doc_id=f"wiki:{md.stem}",
            source="wikipedia:superconductors",
            text=md.read_text(encoding="utf-8"),
            metadata={"title": md.stem.replace("_", " ")},
        )

load_sec_10k()

Palantir Technologies 2024 SEC 10-K, plain text extract.

Yields one Document per section heading. Good for hybrid search and sub-question decomposition (the document is full of numbered items and tables).

Source code in cookbook/corpora.py 
66
67
68
69
70
71
72
73
74
def load_sec_10k() -> Iterator[Document]:
    """Palantir Technologies 2024 SEC 10-K, plain text extract.

    Yields one Document per section heading. Good for hybrid search and
    sub-question decomposition (the document is full of numbered items and
    tables).
    """
    txt = CORPUS_DIR / "sec-10k-PLTR-2024.txt"
    yield from _yield_text_sections(txt, source="sec:PLTR-2024-10K")

load_rust_book()

A curated subset of The Rust Programming Language (CC-BY).

One Document per chapter, with the chapter heading in metadata. Good for code-aware chunking and step-back reasoning.

Source code in cookbook/corpora.py 
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
def load_rust_book() -> Iterator[Document]:
    """A curated subset of *The Rust Programming Language* (CC-BY).

    One Document per chapter, with the chapter heading in metadata. Good for
    code-aware chunking and step-back reasoning.
    """
    folder = CORPUS_DIR / "tech-docs-rust-book"
    for md in sorted(folder.glob("*.md")):
        title_line = md.read_text(encoding="utf-8").splitlines()[0].lstrip("# ").strip()
        yield Document(
            doc_id=f"rust-book:{md.stem}",
            source="cc-by:rust-book",
            text=md.read_text(encoding="utf-8"),
            metadata={"chapter": md.stem, "title": title_line},
        )

load_eval_questions()

The ~80-pair hand-curated evaluation set used by recipes 37–40.

Source code in cookbook/corpora.py 
94
95
96
97
def load_eval_questions() -> list[dict]:
    """The ~80-pair hand-curated evaluation set used by recipes 37–40."""
    qa = CORPUS_DIR / "eval" / "ragas-questions.jsonl"
    return [json.loads(line) for line in qa.read_text(encoding="utf-8").splitlines() if line.strip()]