# Corpus engineering, document intelligence, chunking, and indexes

Most RAG failures attributed to “the model” begin before retrieval: a connector
missed a source, a parser destroyed reading order, a chunk boundary separated a
condition from its conclusion, an ACL was dropped, an embedding migration mixed
spaces, or a deletion never propagated to summaries and caches. This chapter
specifies the entire offline and nearline data plane.

## 1. The corpus is a versioned product

A corpus should be represented as an auditable sequence of states, not a folder
of strings. A minimal source record is

\[
d=(id, source, source\_version, bytes\_hash, observed\_at,
valid\_from, valid\_to, acl, license, parser\_version, metadata, content).
\]

The fields have different meanings:

- `id` is stable across harmless reprocessing.
- `source_version` identifies the upstream revision, ETag, commit, filing, or
  snapshot.
- `bytes_hash` detects byte-level identity and supports chain of custody.
- `observed_at` records when the pipeline saw the content.
- `valid_from` and `valid_to` describe when the fact or document is true in the
  represented world. This is not the same as ingestion time.
- `acl` and classification labels travel with every derived unit.
- `license` and retention policy determine whether content may be indexed,
  transformed, quoted, logged, or used for training.
- `parser_version` makes derived text reproducible.

Use content-addressed immutable raw objects plus a manifest that maps logical
source IDs to versions. Derived artifacts—text blocks, chunks, embeddings,
summaries, graph edges, qrels—reference both the logical ID and exact source
version. Never make the vector-store row the only surviving copy of provenance.

### Snapshot, stream, and bitemporal modes

**Snapshot ingestion** freezes a corpus for experiments and regulated releases.
It is easy to reproduce but becomes stale. **Streaming/change-data capture**
keeps an operational index current but must handle ordering, retries, duplicate
events, tombstones, and partial failure. **Bitemporal storage** records both
valid time and system time so the system can answer “what was true on date X?”
and “what did our system know on date Y?”

An index release manifest should contain:

```text
corpus_snapshot_id
source connector versions and watermarks
raw object hashes
parser/OCR/layout model and configuration
normalization and dedup versions
chunker and enrichment versions
embedding model/tokenizer/pooling/dimension/normalization
index algorithm and parameters
document and chunk counts
ACL/tenant partition policy
build start/end and validation report
parent index and deleted IDs
```

## 2. Acquisition and connector correctness

Connectors are part of answer quality. For each source define:

- discovery method and scope;
- authentication and least-privilege credential;
- pagination and checkpoint semantics;
- full-scan and incremental watermarks;
- rate-limit, retry, and backoff behavior;
- update and deletion signals;
- attachment, linked-resource, and permission traversal;
- regional/data-residency constraints;
- maximum acceptable ingestion lag;
- reconciliation procedure against an authoritative item count.

Exactly-once ingestion is rarely available. Make processing idempotent with a
key such as `(source_id, source_version, transform_version)`. An event is not
complete until raw bytes, manifest state, derived artifacts, and index state
agree. Maintain a dead-letter queue with reason, source, retry count, and last
error; silently skipping malformed files creates a biased knowledge base.

### Web acquisition

Respect robots, terms, copyright, authentication, and canonical URLs. Preserve
HTTP status, final URL, redirect chain, content type, language, crawl time,
ETag/Last-Modified, and response hash. Boilerplate removal can discard important
navigation or legal qualifiers, so keep raw HTML and a block tree. Detect
soft-404 pages, templated near-duplicates, infinite calendars, session IDs, and
rendered content that differs from initial HTML.

### Enterprise and repository acquisition

For document systems, preserve workspace/site/project, owners, sharing groups,
labels, folder path, comments if in scope, and inherited permissions. For code,
index an exact commit and include language, path, symbol table, imports,
references, tests, build metadata, generated/vendor status, and license. For
databases, prefer governed read replicas or CDC and preserve schema/type rather
than serializing every row into prose.

## 3. Document intelligence: bytes are not text

### 3.1 Plain text, HTML, and markup

Retain the DOM or syntax tree long enough to identify headings, paragraphs,
lists, tables, code, quotes, captions, links, and hidden/boilerplate nodes.
Normalize Unicode deliberately (often NFC), but keep an offset map from
normalized text to raw bytes. Do not lowercase or strip punctuation in the
canonical evidence copy; those may carry identifiers, negation, formulas, and
legal meaning. Retrieval-specific normalized views can be derived separately.

### 3.2 PDF

PDF stores positioned glyphs and drawing commands. A robust pipeline needs:

1. file validation, encryption/password handling, and malware-safe isolation;
2. page rendering and native text extraction;
3. OCR when native text is absent or corrupt;
4. layout region detection;
5. reading-order reconstruction;
6. heading/list/table/formula/figure/caption recognition;
7. repeated header/footer and page-number handling;
8. block-to-page bounding boxes and character-offset mapping;
9. confidence and parser provenance per block.

[Nougat](https://arxiv.org/abs/2308.13418) treats scientific document parsing
as image-to-markup generation and is useful for formulas, but no parser is
universally reliable across scans, patents, financial reports, slides, and
multilingual forms. The 2026 study [When Good OCR Is Not
Enough](https://aclanthology.org/2026.acl-industry.60/) shows why character-level
OCR metrics can fail to predict RAG behavior: structural and semantic errors can
break retrieval even when word/character error appears good. Evaluate parsers
through retrieval and answer tasks in addition to OCR/layout scores.

### 3.3 Tables

Store a structured table object:

```text
table_id, document_id, page/region, caption, headers, hierarchical headers,
rows, cell spans, types, units, footnotes, source offsets, extraction confidence
```

Create multiple retrieval views without losing the canonical structure:

- table/schema summary for table-level retrieval;
- row strings that repeat necessary headers;
- column/statistical profiles;
- individual cell neighborhoods for exact lookup;
- graph edges between headers, rows, entities, and referenced tables.

Merged cells, multirow headers, footnotes, blank-as-ditto conventions, units,
and dates are common failure modes. Flattened Markdown is convenient for a
generator but should not be the only stored representation.

### 3.4 Figures, charts, and formulas

Preserve page/region image, nearby caption, referenced text, OCR, extracted data
when available, and modality-specific embeddings. A caption-only index cannot
answer a question about an unlabeled visual trend. A chart pipeline may combine
visual retrieval with chart-to-table extraction and executable calculation.
Generated descriptions are derived evidence and can hallucinate; cite the
original region and record the describing model/version.

### 3.5 Audio and video

Store timestamped transcript segments, speaker labels, language, diarization
confidence, scenes/frames, OCR overlays, captions, and synchronization links.
Chunk boundaries should respect speaker turns or topic segments. Citations need
time ranges, not merely a media URL.

### 3.6 Source code

Parse with a language-aware syntax tree. Retrieval units include symbols,
functions, classes, modules, configuration blocks, tests, issues, commits, and
diff hunks. Preserve exact revision and byte offsets. Build call/import/reference
graphs and pair code with docstrings and tests. Generated or vendored files are
often downweighted or excluded. Secrets must be detected before remote embedding
or logging.

## 4. Normalization without evidence destruction

Maintain three layers:

1. **raw:** original bytes and source response;
2. **canonical:** faithful structured representation with offsets;
3. **retrieval views:** normalized strings, expansions, summaries, embeddings,
   graph nodes, and modality views.

Typical canonicalization includes Unicode normalization, whitespace repair,
hyphenation repair across line breaks, ligature expansion, and deterministic
encoding. Each transformation should emit an alignment map. Aggressive actions
such as stopword deletion, stemming, lowercasing, transliteration, or table
flattening belong in a retrieval view, not the canonical evidence.

Language detection should operate at document and, when needed, block level.
Mixed-language documents and code-switching are common. Tokenizer choice affects
chunk length, embedding truncation, BM25 terms, and cost; record the exact
tokenizer revision.

## 5. Deduplication and contamination control

Duplicates waste index and prompt budget, distort rankings, and let one source
appear independently corroborated. Use several layers:

- exact raw and normalized hashes;
- canonical URL/source identity;
- shingled MinHash or SimHash for near-duplicate prose;
- template/boilerplate fingerprints;
- embedding similarity only as a candidate signal;
- table/image perceptual hashes where appropriate;
- version lineage for legitimate revisions.

Do not simply delete every near-duplicate. Cluster and select a canonical member
using authority, recency, completeness, and license, while retaining lineage.
Distinct official versions or independently authored sources may be semantically
similar but operationally important. During evaluation, prevent train/test,
query/corpus, and benchmark contamination; a retrieved copy of the reference
answer can create misleadingly perfect results.

At answer time, collapse repeated chunks from the same duplicate cluster and do
not count them as source diversity. Poisoning defenses also benefit from cluster
awareness because an attacker can flood the index with paraphrases.

## 6. Metadata and provenance model

Useful metadata is not decoration; it is a retrieval and policy signal.

### Identity and structure

- document, version, block, parent, section, page, region, row, and chunk IDs;
- title and hierarchical heading path;
- exact character/token/page/time offsets;
- source URI and immutable content hash;
- links, citations, references, and entity IDs.

### Time

- authored/published/effective/updated/observed/indexed times;
- valid-from/valid-to;
- supersedes/superseded-by;
- temporal precision and timezone.

### Trust and governance

- source owner/publisher and authority tier;
- tenant, ACL principals/groups, classification, residency;
- license, retention, legal hold, consent, and training eligibility;
- parse/extraction confidence and generated-derivative marker;
- content signature and validation status.

### Retrieval features

- language, domain, document type, product/version, geography;
- entities and aliases;
- quality, popularity, authority, freshness, and duplicate cluster;
- token counts and modality;
- embedding/index version.

Metadata filters should be compiled from explicit user intent and policy. A
filter applied after ANN search can lose all accessible relevant evidence if the
candidate pool was dominated by inaccessible items; prefer pre-filtering,
partitioned search, or filter-aware ANN, followed by a second authorization
check.

## 7. Chunking as an information-preservation problem

Let a document contain evidence spans \(E=\{e_1,\ldots,e_n\}\), and let a
chunker produce units \(U\). Chunking should maximize evidence containment and
retrievability while limiting noise and storage:

\[
\max_U\;\alpha\,\mathrm{containment}(E,U)
+\beta\,\mathrm{retrievability}(U)
-\gamma\,\mathrm{redundancy}(U)
-\delta\,\mathrm{cost}(U).
\]

Because unknown future questions define \(E\), no offline chunker can optimize
this exactly. Evaluate representative questions and evidence spans.

### 7.1 Fixed token windows

Split every \(w\) tokens with overlap \(o\). This is deterministic, cheap, and
batch-friendly. It can cut sentences, lists, definitions, tables, or a condition
from its consequence. Overlap improves boundary coverage but inflates storage,
duplicate retrieval, and prompt redundancy approximately by
\(w/(w-o)\).

Use tokenizer-aware offsets. Do not assume characters, whitespace words, and
model tokens are interchangeable. Ensure units do not exceed embedding-model
limits after adding title and metadata prefixes.

### 7.2 Structure-aware recursive segmentation

Prefer document boundaries in order: section, subsection, paragraph, sentence,
then token fallback. Keep heading paths. This is a strong baseline for manuals,
policies, and Markdown/HTML. Its quality depends on parsing; a PDF with wrong
reading order produces confidently wrong chunks.

### 7.3 Sentence-window retrieval

Index a central sentence or short unit together with surrounding context in the
representation; retrieve the central unit and expand to its window for the
generator. This separates precise matching from coherent reading. Overlapping
windows require deduplication, and a fixed window can still miss distant
definitions.

### 7.4 Semantic segmentation

Embed adjacent sentences and create a boundary when topic distance exceeds a
threshold, or ask a model to identify coherent blocks. Thresholds must be
calibrated by domain and document genre. Embedding drift changes chunk
boundaries; record the model. LLM segmentation adds cost, nondeterminism, and a
new instruction-injection surface if document text is placed in a privileged
prompt.

### 7.5 Proposition indexing

[Dense X Retrieval](https://aclanthology.org/2024.emnlp-main.845/) decomposes
passages into atomic propositions, giving fine-grained units that can match one
fact cleanly. Advantages are precision, multi-hop composition, and claim-level
provenance. Risks include extraction errors, lost qualifiers, pronoun/entity
resolution mistakes, high unit counts, and inability to understand a
proposition without its source context. Store proposition-to-span alignment and
return parent context for generation.

### 7.6 Parent-child and small-to-big retrieval

Index child units for precise matching, but attach a parent section or document
window after retrieval. If several children share a parent, merge them before
packing. Tune child size, parent size, maximum parents, and expansion direction.
This is often more robust than choosing one compromise chunk size.

### 7.7 Contextual prefixes and contextual embeddings

Prefix a chunk with title, heading path, a short document summary, or generated
context before sparse/dense indexing. This can resolve ambiguous local text but
may cause every chunk to match generic summary terms and makes generated
context a potential error source. Keep raw chunk text separate and test prefix
ablation.

Late chunking encodes a longer document first and pools token states over later
chunk spans, allowing each chunk vector to reflect document-wide context. It is
limited by the encoder’s effective context and requires careful offset/pooling
alignment. For extremely long inputs, contextual signals can dilute.

### 7.8 Hierarchical and recursive units

[RAPTOR](https://openreview.net/forum?id=GN921JHCRw) embeds leaf chunks,
clusters them, summarizes clusters, and recursively builds a tree. Retrieval can
select leaf facts or higher abstractions. Hierarchies support corpus- or
document-level synthesis but introduce summary omission/hallucination, large
build cost, and update propagation. Store descendant links and never present a
generated summary as if it were primary evidence.

### 7.9 Modality-aware units

Tables, code, pages, image regions, charts, and transcript turns require native
units. The same document can have multiple coordinated indexes: page image,
parsed paragraph, table row, and figure region. Result fusion should preserve
cross-view identity so the generator does not receive four duplicates of one
page.

## 8. Chunk evaluation

Evaluate chunkers with:

- **evidence containment:** fraction of gold evidence spans wholly or
  sufficiently represented in at least one unit;
- **boundary loss:** gold spans split across units without a retrievable parent;
- **retrieval recall/precision:** under fixed retriever and candidate budget;
- **answer quality:** under fixed selector/generator;
- **redundancy:** duplicate token ratio in retrieved and packed context;
- **unit count/index bytes:** storage and build cost;
- **update amplification:** units and derived summaries changed per source edit;
- **citation localization:** ability to map answer claims back to exact spans;
- **latency and prompt tokens:** at equal answer quality.

A good chunker on Natural Questions may fail for contracts or tables. Build a
gold boundary set from the target corpus, including long evidence, definitions,
lists, exceptions, cross-section references, and multi-page tables.

## 9. Enrichment: useful views with explicit lineage

Common enrichments are titles/headings, keywords, entities, aliases, document
summaries, hypothetical questions, propositions, triples, captions, table
summaries, and trust/freshness scores. Treat every enrichment as a derived
artifact with generator/model version, prompt hash, source spans, confidence,
and status.

Enrichment helps only if it changes retrieval or selection beneficially.
Measure each independently. Synthetic questions can improve recall but bias the
index toward expected phrasing. Entity linking supports graph retrieval but
incorrectly merged entities create false paths. Abstractive summaries can make
global retrieval easier while hiding rare facts.

Never allow document text to instruct the enrichment agent to change policy or
call tools. Place untrusted text in a data-delimited context and validate output
against a schema.

## 10. Sparse index construction

An inverted index maps each term to a postings list of documents/chunks,
frequencies, positions, and optional learned impact scores. Index-time choices
include analyzer, tokenizer, stemming, stopwords, fields, field boosts,
positions, payloads, and compression.

For BM25,

\[
s(q,d)=\sum_{t\in q}\log\!\left(1+\frac{N-n_t+0.5}{n_t+0.5}\right)
\frac{f_{td}(k_1+1)}{f_{td}+k_1(1-b+b|d|/\overline{|d|})}.
\]

`k1` controls term-frequency saturation and `b` length normalization. Tune them
on the actual unit distribution. A fielded index can separately score title,
heading, body, anchors, identifiers, and generated expansions. Filters should
use exact keyword/date/numeric fields rather than analyzed text.

Postings are commonly compressed with gap encoding and block methods. Query
execution can use document-at-a-time or term-at-a-time processing; WAND and
block-max WAND skip documents whose score upper bound cannot enter the current
top-k. Learned sparse models assign impacts to a larger expanded vocabulary,
making index sparsity and query execution central training constraints.

## 11. Dense representation materialization

The embedding contract includes:

```text
model and weights revision
tokenizer and max input length
query/document instruction or prefix
pooling and normalization
output dimension and numeric type
distance metric
quantization/truncation
source and chunk transform versions
```

Cosine on normalized vectors equals dot-product ranking; mixing normalized and
unnormalized vectors changes results. Query and document towers may require
different prefixes. Instruction-conditioned document embeddings can require
re-encoding for a new instruction, while some methods transform a generic
space. Truncation must be detected—not silently accepted.

Batch by token count, record failures, and verify deterministic input ordering.
Maintain row-to-chunk identity outside the vector payload. Sample vector norms,
NaNs, duplicate vectors, and language/domain distributions. A successful HTTP
embedding call does not prove a valid index.

## 12. Exact dense search as an oracle

For matrix \(D\in\mathbb R^{N\times m}\) and query \(q\), exact inner-product
search computes \(Dq\) and sorts/selects the largest scores. It costs
\(O(Nm)\) per query but is essential on a representative slice to measure ANN
loss. GPU matrix multiplication can make exact search practical for smaller
corpora and batched queries.

Approximate search quality is often measured as recall of exact top-k, but also
measure task qrel recall and final answer impact. Vector-neighbor identity is a
proxy, not the product objective.

## 13. ANN index algorithms

### 13.1 Locality-sensitive hashing

LSH draws hash functions such that nearby points collide with higher
probability. Multiple tables and probes improve recall at storage/query cost.
It offers analyzable probability guarantees but may need many candidates for
modern high-dimensional semantic spaces.

### 13.2 Coarse quantization and IVF

Cluster vectors into \(K\) coarse centroids and assign each vector to one or
more posting lists. At query time search the `nprobe` closest centroids and
score only their residents. Larger `nprobe` improves recall and increases work.
Unbalanced clusters create tail latency; training data must represent the
deployed vector distribution.

### 13.3 Product quantization

Split an \(m\)-dimensional vector into \(M\) subvectors and replace each with a
codebook index. Approximate distance uses precomputed query-to-codeword tables.
PQ can reduce each vector from hundreds/thousands of bytes to tens of bytes, but
distorts distances. Optimized/residual PQ rotates or quantizes residuals. Always
report code size, training sample, reconstruction error, neighbor recall, and
answer impact.

### 13.4 HNSW

Hierarchical Navigable Small World graphs assign points to random levels. Upper
levels provide long-range navigation; level zero supplies dense local search.
Key parameters include `M` (edges), `efConstruction`, and query `efSearch`.
Higher values generally improve recall while increasing build time, memory, and
latency. Deletes and frequent updates can degrade graph quality depending on the
implementation; periodically measure from a rebuilt baseline.

HNSW is an empirical workhorse, not a universal theoretical guarantee. A 2023
[worst-case analysis](https://proceedings.neurips.cc/paper_files/paper/2023/hash/d0ac28b79816b51124fcc804b2496a36-Abstract-Conference.html)
constructs cases requiring linear exploration for common graph indexes. Test on
the deployed distribution and filters.

### 13.5 DiskANN/Vamana

[DiskANN](https://proceedings.neurips.cc/paper/2019/hash/09853c7fb1d3f8ee67a61b6bf4a7f8e6-Abstract.html)
uses a pruned navigable graph designed for SSD, compressed vectors in memory,
and beam/batched I/O. Its paper reports billion-point search on a single node;
those hardware-specific results do not transfer automatically to text
embeddings. Disk layout, cache warmness, queue depth, and tail latency matter.

### 13.6 SPANN and hybrid memory/disk layouts

[SPANN](https://proceedings.neurips.cc/paper_files/paper/2021/hash/299dc35e747eb77177d9cea10a802da2-Abstract.html)
stores centroids in memory and posting lists on disk, adds points to neighboring
cluster closures, and prunes lists at query time. It illustrates a general
systems principle: index quality depends on memory hierarchy and I/O pattern,
not only a mathematical neighbor algorithm.

### 13.7 ScaNN and anisotropic quantization

ScaNN combines partitioning, quantization optimized for maximum inner product,
and reordering/exact rescoring. MIPS errors parallel to a query can affect
ranking more than orthogonal reconstruction error; quantization objectives can
reflect that. Benchmark with the same distance, dimension, batch size, and
hardware intended for production.

### 13.8 Multi-vector search

Late-interaction models may store dozens or hundreds of token/patch vectors per
unit. Practical systems use centroid assignment, residual compression,
inverted lists, query-token candidate generation, MaxSim aggregation, and
candidate reranking. Page-image systems face particularly large storage. Report
vectors per unit, bytes per unit, candidate stages, and query latency—not just
embedding dimension.

## 14. Metadata filtering and multitenancy

There are four common patterns:

1. **Physical partition:** separate index per tenant/security domain. Strong
   isolation, operational explosion for many small tenants.
2. **Filter-aware shared index:** candidates are generated under an ACL/filter.
   Efficient if the index supports selective filters well.
3. **Over-retrieve then filter:** easy but can destroy recall for selective
   filters and can leak scores/counts if not carefully isolated.
4. **Global public plus private overlays:** search a common index and authorized
   tenant/user indexes, then fuse results.

Authorization uses immutable principal/group IDs, not names. Resolve membership
at request time or a well-defined policy snapshot. Apply policy before content
is sent to a reranker, generator, external API, cache, or log. Recheck at output
because policy can change during long-running agents. Do not expose that an
inaccessible document exists through snippets, timing, result counts, or
fallback wording.

Filtered ANN can have severe recall/latency interactions: a global graph may
route through disallowed nodes, while post-filtering may yield fewer than k.
Benchmark filter selectivity slices and worst-case tenants.

## 15. Freshness, updates, and deletion

Define service-level objectives for:

- source change to raw capture;
- raw capture to parsed canonical state;
- canonical state to sparse/dense/graph indexes;
- index availability across replicas;
- cache invalidation;
- deletion across raw, derived, index, cache, log, and backup layers.

Use tombstones with monotonically ordered source versions. An older delayed
update must not resurrect deleted content. Keep index generation IDs in every
answer trace. Mixed-generation reads can combine incompatible chunk or
embedding versions.

For mutable facts, do not rely only on recency decay. Some questions ask for
historical state; some old authoritative documents remain valid; some new
documents quote outdated facts. Extract or attach validity intervals and group
contradictory versions. Rank by temporal compatibility with the query, source
authority, and update status.

Deletion includes:

- source document and raw object;
- normalized blocks and chunks;
- sparse postings and vector rows;
- generated summaries, questions, propositions, captions, and translations;
- entity/graph nodes and edges derived only from the source;
- caches, evaluation traces, fine-tuning datasets, and logs;
- replicas and backups under the declared retention schedule.

Maintain reverse lineage from source to every derivative so deletion is
computable. Otherwise “remove document from vector DB” is not a deletion
guarantee.

## 16. Index migration and compatibility

Changing embedding weights, tokenizer, pooling, instructions, dimension,
normalization, chunking, or corpus changes the index generation. A safe
migration is:

1. build a new immutable generation;
2. validate counts, hashes, vector statistics, ACL parity, exact/ANN recall,
   qrels, answer quality, latency, and cost;
3. shadow or dual-read representative traffic;
4. compare paired per-query outputs and failure slices;
5. gradually route traffic with rollback;
6. retain the old generation until audit and deletion windows permit removal.

Never query old document vectors with a new incompatible query encoder. If a
model claims backward compatibility, verify cross-version retrieval directly.
Aliases should point atomically to a complete generation, not a half-built
collection.

## 17. Quality gates for an index release

### Completeness

- authoritative source counts reconcile;
- no unexplained parse/embedding/index failures;
- all current IDs present and tombstoned IDs absent;
- parent/child and cross-view links resolve;
- expected languages, domains, dates, and tenants represented.

### Correctness

- sample raw-to-canonical and canonical-to-chunk alignment;
- page/region/time offsets resolve to original evidence;
- tables retain headers, units, and row identity;
- ACL checks match the source of truth;
- exact and ANN search return expected seeded cases;
- filters and temporal queries behave at high selectivity.

### Retrieval

- qrel Recall@k/nDCG/MRR by slice;
- exact sparse/dense/hybrid and ANN comparisons;
- boundary, identifier, paraphrase, multi-hop, table, visual, and stale cases;
- duplicate and poisoned-cluster tests;
- result stability and score distributions.

### Systems

- build/update throughput, lag, memory, disk, and replication;
- cold/warm p50/p95/p99 latency and saturation;
- degradation under deletes, filters, concurrent updates, and node loss;
- rollback and disaster-recovery rehearsal.

### Governance

- source license/consent/retention completeness;
- no unauthorized cross-tenant retrieval in adversarial tests;
- deletion and legal-hold behavior verified;
- data/model/version manifest signed and stored.

## 18. Practical design recipes

### Small, curated text corpus

Use structure-aware paragraphs with heading prefixes, BM25 plus exact dense
search, metadata filters, and parent expansion. Exact search may be simpler and
more reliable than ANN. Deduplicate and keep complete source offsets.

### Large mutable enterprise corpus

Use CDC plus reconciliation, immutable generation manifests, document-type
specific parsers, parent-child units, BM25 and filter-aware ANN, per-domain or
tenant overlays, reranking, blue/green migrations, and end-to-end deletion
lineage. Measure ingestion lag and ACL correctness as first-class SLOs.

### Scientific PDF collection

Keep native text and page renderings, layout/formula/table extraction, section
hierarchy, reference graph, paragraph/proposition units, and visual page index.
Route text questions to hybrid retrieval and visual/layout questions to a visual
retriever. Cite pages/regions and store parser confidence.

### Frequently changing facts

Use live APIs or tightly monitored CDC, valid-time metadata, contradiction
groups, short caches keyed by index generation and as-of time, and replayable
response snapshots. Test stale-answer rate rather than assuming retrieval
equals freshness.

### High-security multitenant corpus

Partition trust domains where feasible; enforce ACL before ANN/reranking;
isolate embedding and generation services; redact or locally process secrets;
disable cross-tenant semantic caches; preserve immutable access/audit logs;
continuously test leakage, membership inference, prompt injection, and deletion.

## 19. What the executable notebooks model

The corpus notebook implements deterministic normalization, stable hashes,
exact and near-duplicate clustering, fixed/structure/sentence/parent-child
units, lineage manifests, access-control filtering, and update/tombstone
behavior on a small corpus. The index notebook compares exact search, an
inverted index, IVF-style candidate restriction, quantization error, and ANN
recall conceptually in standard Python.

These examples teach invariants and measurement. They do not reproduce the
distributed engineering, learned parsers, billion-vector indexes, or model
quality of production systems. Their purpose is to make hidden data-plane
choices observable and testable.