Building the Library
The working page
RAG quality is bounded before a query arrives. Parsing, canonicalization, source identity, permissions, chunk lineage, index construction, update semantics, and deletion decide which evidence can ever be found. This lab replaces the simplistic “load documents into a vector database” story with an auditable corpus pipeline.
Learning outcomes
- distinguish logical document IDs, versions, content hashes, chunks, and index releases;
- diagnose parser and chunker loss separately from retrieval loss;
- build fixed-window, sentence, section, and parent–child representations;
- inspect a postings-list BM25 index rather than scanning every document;
- compare exact cosine search with an inverted-file (IVF) ANN index;
- measure ANN recall and vector-quantization distortion;
- enforce authorization before top-k selection and propagate tombstones.
Companion chapter: Corpus engineering and indexes.
from pathlib import Path
import sys
ROOT = Path.cwd()
if not (ROOT / "src").exists():
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / "src"))
print("Repository root: resolved from the notebook location")
Repository root: resolved from the notebook location
1. The corpus is a versioned data product
A production evidence unit needs a stable source ID, source/version time, parser version, content hash, permission policy, trust domain, language, media type, and lineage back to exact characters or regions. The logical ID answers “which source is this?”; the content hash answers “which bytes/text did this index use?”; the snapshot hash answers “which set of versions did this release contain?” These are not interchangeable.
Ingestion should be replayable and idempotent. Change-data capture creates new versions; it does not silently overwrite history. Exact duplicates may be suppressed safely. Near duplicates need an explicit policy because two similar documents can be independent corroboration, a syndicated copy, or a poisoning amplification cluster.
from rag_evolution.ingestion import ACLPolicy, CorpusManifest
from rag_evolution.models import Document
documents = (
Document(
id="retrieval-guide",
title="Retrieval Guide",
source="https://example.test/retrieval",
date="2026-08-01",
text=(
"# Sparse retrieval\nAn inverted index stores postings for lexical search. "
"BM25 saturates term frequency and normalizes document length.\n\n"
"## Dense retrieval\nA dual encoder maps queries and passages into vectors. "
"Approximate nearest-neighbor indexes trade recall for latency."
),
),
Document(
id="grounding-guide",
title="Grounding Guide",
source="https://example.test/grounding",
date="2026-08-02",
text=(
"# Evidence\nAnswers should map atomic claims to immutable source spans. "
"Citation syntax alone does not establish entailment or authority.\n\n"
"## Abstention\nThe system should abstain when evidence is absent or conflicting."
),
),
Document(
id="security-guide",
title="Security Guide",
source="https://example.test/security",
date="2026-08-03",
text=(
"# Trust boundary\nRetrieved text is untrusted data, not executable instruction. "
"Tenant and row permissions must be applied before ranking.\n\n"
"## Deletion\nTombstones must reach chunks, indexes, graphs, caches, and backups."
),
),
)
manifest = CorpusManifest(near_duplicate_threshold=0.72, shingle_width=3)
policies = {
"retrieval-guide": ACLPolicy.public(),
"grounding-guide": ACLPolicy.restricted(("group:research",)),
"security-guide": ACLPolicy.public(),
}
decisions = [manifest.ingest(doc, acl=policies[doc.id]) for doc in documents]
print("Actions:", [(item.entry.document_id, item.action) for item in decisions if item.entry])
print("Snapshot:", manifest.snapshot_hash()[:16])
print("Anonymous visibility:", [entry.document_id for entry in manifest.active_entries(("anonymous",))])
print("Research visibility:", [entry.document_id for entry in manifest.active_entries(("group:research",))])
Actions: [('retrieval-guide', 'created'), ('grounding-guide', 'created'), ('security-guide', 'created')]
Snapshot: 2b9b473d9767cc2b
Anonymous visibility: ['retrieval-guide', 'security-guide']
Research visibility: ['grounding-guide', 'retrieval-guide', 'security-guide']
2. Parsing is model construction, not clerical cleanup
HTML requires boilerplate removal, canonical URL handling, DOM structure, table and list preservation, and defenses against hidden/active content. PDFs require reading-order recovery, headers/footers, columns, equations, footnotes, tables, figures, OCR confidence, and page coordinates. Slides, spreadsheets, code, audio, video, and scanned forms each need different structural units. A parser can produce fluent but wrong text: a transposed table, detached caption, or reordered two-column page may be impossible for a downstream retriever to repair.
A parser benchmark therefore needs element-level precision/recall, reading order, table cell fidelity, equation fidelity, OCR character error, source coordinates, latency, and cost—not only whether output text exists. Keep original bytes and parser artifacts beside normalized text.
from rag_evolution.ingestion import canonicalize_text, content_hash
noisy = " Retrieval\r\n\r\nuses\tpostings. "
canonical = canonicalize_text(noisy)
duplicate = manifest.ingest(
Document(
id="retrieval-copy",
text=documents[0].text,
title=documents[0].title,
source="https://mirror.test/retrieval",
)
)
near = manifest.ingest(
Document(
id="retrieval-near-copy",
text=documents[0].text.replace("latency", "speed"),
title="Syndicated retrieval guide",
)
)
print("Canonical:", repr(canonical))
print("Hash:", content_hash(noisy)[:16])
print("Exact duplicate:", duplicate.action, duplicate.duplicate_of)
print("Near duplicate:", near.action, near.duplicate_of, round(near.similarity, 3))
Canonical: 'Retrieval\n\nuses postings.' Hash: e76a6f21dddb1113 Exact duplicate: exact_duplicate retrieval-guide Near duplicate: near_duplicate retrieval-guide 0.943
3. Updates, permissions, and deletion are index semantics
Permission filtering after ANN top-k can return too few results and may leak scores, cache entries, or timing. Filter-aware indexes, tenant partitions, or oversampling plus a verified post-filter are design choices that must be evaluated under realistic ACL selectivity. The safe contract preserves ACL metadata through parsing, chunking, candidate generation, reranking, generation, citations, logging, and caching.
Deletion is likewise end to end. A source tombstone must invalidate every derived child chunk, embedding, postings entry, graph node/edge, summary, answer cache, training export, and replica. “Removed from the UI” is not an unlearning guarantee.
updated = manifest.ingest(
Document(
id="security-guide",
title="Security Guide",
source="https://example.test/security",
date="2026-08-04",
text=documents[2].text + "\n\n## Audit\nEvery release records immutable content hashes.",
),
acl=ACLPolicy.public(),
)
before_delete = manifest.snapshot_hash()
tombstone = manifest.tombstone("grounding-guide", "source owner requested deletion")
print("Update:", updated.action, "version", updated.entry.version)
print("Tombstone:", tombstone.document_id, tombstone.version, tombstone.tombstone_reason)
print("Snapshot changed:", before_delete != manifest.snapshot_hash())
print("History versions:", [entry.version for entry in manifest.history("grounding-guide")])
Update: updated version 2 Tombstone: grounding-guide 2 source owner requested deletion Snapshot changed: True History versions: [1, 2]
4. Chunking defines the retrieval hypothesis space
Common families include fixed token windows, sentences, paragraphs, Markdown/DOM sections, recursive separator splitting, discourse units, semantic-boundary segmentation, propositions, parent–child indexes, late chunking after long-document encoding, hierarchical summaries, table rows/regions, code symbols, graph nodes, and page-image patches.
Smaller units improve localization and reduce distractors but lose context. Larger units preserve discourse but dilute similarity and consume the prompt. Overlap improves boundary recall while inflating storage, correlated candidates, and citation ambiguity. Tune chunking jointly with retriever, reranker, top-k, generator, and task; do not optimize a universal chunk size.
from rag_evolution.chunking import section_chunks, sentence_chunks
sentence_view = sentence_chunks(documents[0], max_tokens=14, overlap_sentences=1)
section_view = section_chunks(documents[0], max_tokens=35)
print("Sentence chunks:")
for item in sentence_view:
print(item.chunk.id, (item.lineage.start_char, item.lineage.end_char), repr(item.chunk.text[:58]))
print("Section paths:")
for item in section_view:
print(item.chunk.id, item.lineage.section_path, item.lineage.end_token - item.lineage.start_token)
Sentence chunks:
retrieval-guide::sentence::0000 (0, 72) '# Sparse retrieval\nAn inverted index stores postings for l'
retrieval-guide::sentence::0001 (19, 72) 'An inverted index stores postings for lexical search.'
retrieval-guide::sentence::0002 (73, 154) 'BM25 saturates term frequency and normalizes document leng'
retrieval-guide::sentence::0003 (136, 209) '## Dense retrieval\nA dual encoder maps queries and passage'
retrieval-guide::sentence::0004 (155, 209) 'A dual encoder maps queries and passages into vectors.'
retrieval-guide::sentence::0005 (210, 272) 'Approximate nearest-neighbor indexes trade recall for late'
Section paths:
retrieval-guide::section::0000 ('Sparse retrieval',) 18
retrieval-guide::section::0001 ('Sparse retrieval', 'Dense retrieval') 18
5. Parent–child retrieval separates search granularity from reading granularity
Retrieve a compact child because it has a sharp signal; send its larger parent because the generator needs definitions, qualifiers, or surrounding table rows. The child must store an exact parent edge and both need source coordinates. Parent expansion can otherwise silently exceed budgets or duplicate the same section several times.
Evaluate child retrieval recall, parent expansion recall, packed evidence recall, duplicate rate, token cost, and citation precision separately.
from rag_evolution.chunking import parent_child_chunks
hierarchy = parent_child_chunks(
documents[0], parent_max_tokens=38, child_max_tokens=14, child_overlap_sentences=0
)
print("Parents:", [(item.chunk.id, item.lineage.section_path) for item in hierarchy.parents])
print("Children -> parent:")
for item in hierarchy.children:
print(item.chunk.id, "->", item.lineage.parent_chunk_id, "chars", (item.lineage.start_char, item.lineage.end_char))
Parents: [('retrieval-guide::parent::0000', ('Sparse retrieval',)), ('retrieval-guide::parent::0001', ('Sparse retrieval', 'Dense retrieval'))]
Children -> parent:
retrieval-guide::child::0000 -> retrieval-guide::parent::0000 chars (2, 72)
retrieval-guide::child::0001 -> retrieval-guide::parent::0000 chars (73, 133)
retrieval-guide::child::0002 -> retrieval-guide::parent::0001 chars (139, 209)
retrieval-guide::child::0003 -> retrieval-guide::parent::0001 chars (210, 271)
6. Sparse retrieval is an execution engine
An inverted index stores a postings list for each term. Query evaluation visits only postings for query terms, accumulates BM25/query-likelihood scores, and uses WAND/Block-Max WAND bounds to avoid fully scoring documents that cannot enter top-k. Fielded BM25, phrase/proximity, analyzers, stemming, multilingual tokenization, spelling, entity aliases, numeric/date handling, and pseudo-relevance feedback remain powerful—especially for identifiers, rare terms, code, names, and fresh vocabulary.
Learned sparse models such as SPLADE retain inverted-index execution while learning expansion and term weights. Their operational questions include posting expansion, index size, latency, regularization, and domain drift.
from rag_evolution.indexes import InvertedIndex
index_chunks = tuple(item.chunk for doc in documents for item in sentence_chunks(doc, max_tokens=20, overlap_sentences=0))
sparse_index = InvertedIndex(index_chunks)
sparse_hits = sparse_index.search("inverted postings BM25 document length", k=4)
print("Vocabulary terms:", len(sparse_index.postings))
print("Postings for 'index':", sparse_index.postings.get("index", ()))
print("Results:", [(hit.chunk.document_id, round(hit.score, 3)) for hit in sparse_hits])
allowed = {chunk.id for chunk in index_chunks if chunk.document_id == "security-guide"}
print("Pre-top-k ACL/filter result:", [hit.chunk.document_id for hit in sparse_index.search("index permissions", 5, allowed)])
Vocabulary terms: 86
Postings for 'index': (Posting(document_ordinal=0, term_frequency=1),)
Results: [('retrieval-guide', 7.03)]
Pre-top-k ACL/filter result: ['security-guide']
7. Dense retrieval needs an embedding contract
Record model and tokenizer revision, pooling, normalization, distance metric, dimensionality, maximum input length, query/document prefixes, truncation, language/domain assumptions, batching precision, and training data. A mismatched prefix or cosine-vs-inner-product setting can invalidate an index without an obvious error.
Exact search is the correctness oracle. Approximate search is a systems optimization and must be evaluated against exact neighbors on the current vector distribution. Retrieval quality additionally requires qrels: ANN recall only says whether the approximate engine reproduced exact embedding neighbors, not whether the embedding put relevant evidence nearby.
from rag_evolution.indexes import ExactCosineIndex
from rag_evolution.text import tokenize
axes = ("sparse", "dense", "evidence", "security")
expansions = {
"sparse": {"sparse", "lexical", "bm25", "postings", "inverted"},
"dense": {"dense", "vector", "encoder", "nearest-neighbor"},
"evidence": {"evidence", "citation", "claim", "entailment"},
"security": {"security", "permission", "tenant", "untrusted", "tombstone"},
}
def lab_vector(text):
terms = set(tokenize(text))
return tuple(float(len(terms & expansions[axis])) for axis in axes)
vectors = tuple(lab_vector(chunk.title + " " + chunk.text) for chunk in index_chunks)
exact = ExactCosineIndex(index_chunks, vectors)
query_vector = lab_vector("dense vector nearest-neighbor encoder")
exact_hits = exact.search(query_vector, k=4)
print("Vector dimensions:", axes)
print("Exact neighbors:", [(hit.chunk.id, round(hit.score, 3)) for hit in exact_hits])
Vector dimensions: ('sparse', 'dense', 'evidence', 'security')
Exact neighbors: [('retrieval-guide::sentence::0001', 1.0), ('retrieval-guide::sentence::0000', 0.196), ('grounding-guide::sentence::0000', 0.0), ('grounding-guide::sentence::0001', 0.0)]
8. ANN families express different resource trade-offs
- HNSW navigates a multilayer proximity graph; it offers strong recall and latency but consumes RAM and has update/filtering considerations.
- IVF learns coarse cells and probes only likely lists;
nlistandnprobecontrol build/search work and recall. - PQ/OPQ compress vectors into subspace codes; asymmetric distance tables trade reconstruction error for memory and bandwidth.
- DiskANN/SPANN organize graph/centroid structures around SSD access for billion-scale collections.
- ScaNN combines partitioning, quantization, and reordering.
- Multi-vector indexes (ColBERT/ColPali) add token/patch-level storage and MaxSim execution; PLAID-style pruning reduces that cost.
Measure recall@k versus exact search, qrels-based nDCG/recall, p50/p95 latency, throughput, RAM/disk, build time, update/delete cost, filter selectivity, and performance under distribution shift.
from rag_evolution.indexes import IVFCoarseIndex, evaluate_ann_recall
ivf = IVFCoarseIndex(index_chunks, vectors, nlist=3, iterations=10)
query_vectors = (
lab_vector("dense vector encoder"),
lab_vector("citation evidence entailment"),
lab_vector("tenant permission security"),
)
for nprobe in range(1, ivf.nlist + 1):
audit = evaluate_ann_recall(exact, ivf, query_vectors, k=3, nprobe=nprobe)
print("nprobe", nprobe, "mean ANN recall@3", round(audit.mean_recall, 3), "per query", audit.per_query)
print("IVF lists:", dict(ivf.inverted_lists))
nprobe 1 mean ANN recall@3 0.667 per query (0.6666666666666666, 0.6666666666666666, 0.6666666666666666)
nprobe 2 mean ANN recall@3 0.889 per query (1.0, 1.0, 0.6666666666666666)
nprobe 3 mean ANN recall@3 1.0 per query (1.0, 1.0, 1.0)
IVF lists: {0: (0, 1), 1: (2, 3), 2: (4, 5)}
9. Compression must include codebooks and distortion
Float16/8-bit scalar quantization, product quantization, binary codes, Matryoshka dimension truncation, pooling, and tiered hot/cold indexes reduce cost differently. Report code bytes and codebook/metadata bytes, build cost, reconstruction error, neighbor recall, downstream answer/citation quality, and hardware. Tiny toy corpora can have compression ratios below one because shared codebooks dominate; scale changes the accounting.
from rag_evolution.indexes import ProductQuantizer, ScalarQuantizer, audit_quantization
scalar = ScalarQuantizer(bits=4).fit(vectors)
product = ProductQuantizer(subquantizers=2, bits=2, iterations=8).fit(vectors)
for name, quantizer in (("4-bit scalar", scalar), ("2x2-bit PQ", product)):
audit = audit_quantization(vectors, quantizer)
print(
name,
{"original": audit.original_bytes, "codes": audit.encoded_bytes,
"codebook": audit.codebook_bytes, "ratio": round(audit.compression_ratio, 3),
"mse": round(audit.mean_squared_error, 4),
"cosine": round(audit.mean_cosine_similarity, 4)},
)
4-bit scalar {'original': 96, 'codes': 12, 'codebook': 32, 'ratio': 2.182, 'mse': 0.0002, 'cosine': 1.0}
2x2-bit PQ {'original': 96, 'codes': 6, 'codebook': 64, 'ratio': 1.371, 'mse': 0.0278, 'cosine': 0.9974}
10. Corpus/index release gate
A releasable index records source snapshots, parser/chunker/embedding/index versions, permissions, exact-vs-ANN audit, duplicate policy, deletion replay, per-language/domain slices, storage, build/update latency, and rollback ID. Test boundary facts split across chunks, tables, captions, OCR corruption, rare identifiers, ACL-selective queries, changed/deleted sources, and adversarial duplicates. Compare chunkers under the same retriever and compare retrievers under the same chunks before claiming causality.
This lab does not reproduce neural encoders, WAND, HNSW, DiskANN, or billion-scale performance. It supplies exact transparent baselines and the measurement contracts required to evaluate those implementations honestly.
from rag_evolution.operations import release_manifest
release = release_manifest(
manifest.snapshot_hash(), "parser-lab-v1", "lineage-chunker-v1",
"lab-vector-v1", "ivf-v1", "none", "none", "none"
)
print("Release ID:", release["release_id"])
print("Components:", {key: value for key, value in release.items() if key != "release_id"})
Release ID: fc5d49618ba0b200225296ad46096dd5973685cb65d00d82c6bb37a7a8299e9d
Components: {'corpus_snapshot': 'abcfca0b7cdd09c1f18430388c6dd27a97c00c59b54007b5815fad02cb80c97b', 'parser_version': 'parser-lab-v1', 'chunker_version': 'lineage-chunker-v1', 'embedding_version': 'lab-vector-v1', 'index_version': 'ivf-v1', 'reranker_version': 'none', 'generator_version': 'none', 'prompt_version': 'none'}