{ "cells": [ { "cell_type": "markdown", "metadata": { "tags": [ "field-notebook-cover", "field-notebook-style" ] }, "source": [ "\n", "\n", "
\n", "
LAB 04 · THE CORPUS
\n", "

Building the Library

\n", "
Lineage, chunking, postings, approximate search, quantization, and deletion
\n", "
quality is bounded before the first query arrives
\n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The working page\n", "\n", "RAG quality is bounded before a query arrives. Parsing, canonicalization,\n", "source identity, permissions, chunk lineage, index construction, update\n", "semantics, and deletion decide which evidence can ever be found. This lab\n", "replaces the simplistic “load documents into a vector database” story with\n", "an auditable corpus pipeline.\n", "\n", "**Learning outcomes**\n", "\n", "- distinguish logical document IDs, versions, content hashes, chunks, and index releases;\n", "- diagnose parser and chunker loss separately from retrieval loss;\n", "- build fixed-window, sentence, section, and parent–child representations;\n", "- inspect a postings-list BM25 index rather than scanning every document;\n", "- compare exact cosine search with an inverted-file (IVF) ANN index;\n", "- measure ANN recall and vector-quantization distortion;\n", "- enforce authorization before top-k selection and propagate tombstones.\n", "\n", "Companion chapter: [Corpus engineering and indexes](../research/corpus_and_indexing.md).\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Repository root: resolved from the notebook location\n" ] } ], "source": [ "from pathlib import Path\n", "import sys\n", "\n", "ROOT = Path.cwd()\n", "if not (ROOT / \"src\").exists():\n", " ROOT = ROOT.parent\n", "sys.path.insert(0, str(ROOT / \"src\"))\n", "print(\"Repository root: resolved from the notebook location\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. The corpus is a versioned data product\n", "\n", "A production evidence unit needs a stable source ID, source/version time,\n", "parser version, content hash, permission policy, trust domain, language,\n", "media type, and lineage back to exact characters or regions. The **logical\n", "ID** answers “which source is this?”; the **content hash** answers “which\n", "bytes/text did this index use?”; the **snapshot hash** answers “which set of\n", "versions did this release contain?” These are not interchangeable.\n", "\n", "Ingestion should be replayable and idempotent. Change-data capture creates\n", "new versions; it does not silently overwrite history. Exact duplicates may\n", "be suppressed safely. Near duplicates need an explicit policy because two\n", "similar documents can be independent corroboration, a syndicated copy, or a\n", "poisoning amplification cluster.\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Actions: [('retrieval-guide', 'created'), ('grounding-guide', 'created'), ('security-guide', 'created')]\n", "Snapshot: 2b9b473d9767cc2b\n", "Anonymous visibility: ['retrieval-guide', 'security-guide']\n", "Research visibility: ['grounding-guide', 'retrieval-guide', 'security-guide']\n" ] } ], "source": [ "from rag_evolution.ingestion import ACLPolicy, CorpusManifest\n", "from rag_evolution.models import Document\n", "\n", "documents = (\n", " Document(\n", " id=\"retrieval-guide\",\n", " title=\"Retrieval Guide\",\n", " source=\"https://example.test/retrieval\",\n", " date=\"2026-08-01\",\n", " text=(\n", " \"# Sparse retrieval\\nAn inverted index stores postings for lexical search. \"\n", " \"BM25 saturates term frequency and normalizes document length.\\n\\n\"\n", " \"## Dense retrieval\\nA dual encoder maps queries and passages into vectors. \"\n", " \"Approximate nearest-neighbor indexes trade recall for latency.\"\n", " ),\n", " ),\n", " Document(\n", " id=\"grounding-guide\",\n", " title=\"Grounding Guide\",\n", " source=\"https://example.test/grounding\",\n", " date=\"2026-08-02\",\n", " text=(\n", " \"# Evidence\\nAnswers should map atomic claims to immutable source spans. \"\n", " \"Citation syntax alone does not establish entailment or authority.\\n\\n\"\n", " \"## Abstention\\nThe system should abstain when evidence is absent or conflicting.\"\n", " ),\n", " ),\n", " Document(\n", " id=\"security-guide\",\n", " title=\"Security Guide\",\n", " source=\"https://example.test/security\",\n", " date=\"2026-08-03\",\n", " text=(\n", " \"# Trust boundary\\nRetrieved text is untrusted data, not executable instruction. \"\n", " \"Tenant and row permissions must be applied before ranking.\\n\\n\"\n", " \"## Deletion\\nTombstones must reach chunks, indexes, graphs, caches, and backups.\"\n", " ),\n", " ),\n", ")\n", "\n", "manifest = CorpusManifest(near_duplicate_threshold=0.72, shingle_width=3)\n", "policies = {\n", " \"retrieval-guide\": ACLPolicy.public(),\n", " \"grounding-guide\": ACLPolicy.restricted((\"group:research\",)),\n", " \"security-guide\": ACLPolicy.public(),\n", "}\n", "decisions = [manifest.ingest(doc, acl=policies[doc.id]) for doc in documents]\n", "print(\"Actions:\", [(item.entry.document_id, item.action) for item in decisions if item.entry])\n", "print(\"Snapshot:\", manifest.snapshot_hash()[:16])\n", "print(\"Anonymous visibility:\", [entry.document_id for entry in manifest.active_entries((\"anonymous\",))])\n", "print(\"Research visibility:\", [entry.document_id for entry in manifest.active_entries((\"group:research\",))])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Parsing is model construction, not clerical cleanup\n", "\n", "HTML requires boilerplate removal, canonical URL handling, DOM structure,\n", "table and list preservation, and defenses against hidden/active content.\n", "PDFs require reading-order recovery, headers/footers, columns, equations,\n", "footnotes, tables, figures, OCR confidence, and page coordinates. Slides,\n", "spreadsheets, code, audio, video, and scanned forms each need different\n", "structural units. A parser can produce fluent but wrong text: a transposed\n", "table, detached caption, or reordered two-column page may be impossible for\n", "a downstream retriever to repair.\n", "\n", "A parser benchmark therefore needs element-level precision/recall, reading\n", "order, table cell fidelity, equation fidelity, OCR character error, source\n", "coordinates, latency, and cost—not only whether output text exists. Keep\n", "original bytes and parser artifacts beside normalized text.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Canonical: 'Retrieval\\n\\nuses postings.'\n", "Hash: e76a6f21dddb1113\n", "Exact duplicate: exact_duplicate retrieval-guide\n", "Near duplicate: near_duplicate retrieval-guide 0.943\n" ] } ], "source": [ "from rag_evolution.ingestion import canonicalize_text, content_hash\n", "\n", "noisy = \" Retrieval\\r\\n\\r\\nuses\\tpostings. \"\n", "canonical = canonicalize_text(noisy)\n", "duplicate = manifest.ingest(\n", " Document(\n", " id=\"retrieval-copy\",\n", " text=documents[0].text,\n", " title=documents[0].title,\n", " source=\"https://mirror.test/retrieval\",\n", " )\n", ")\n", "near = manifest.ingest(\n", " Document(\n", " id=\"retrieval-near-copy\",\n", " text=documents[0].text.replace(\"latency\", \"speed\"),\n", " title=\"Syndicated retrieval guide\",\n", " )\n", ")\n", "print(\"Canonical:\", repr(canonical))\n", "print(\"Hash:\", content_hash(noisy)[:16])\n", "print(\"Exact duplicate:\", duplicate.action, duplicate.duplicate_of)\n", "print(\"Near duplicate:\", near.action, near.duplicate_of, round(near.similarity, 3))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Updates, permissions, and deletion are index semantics\n", "\n", "Permission filtering after ANN top-k can return too few results and may leak\n", "scores, cache entries, or timing. Filter-aware indexes, tenant partitions,\n", "or oversampling plus a verified post-filter are design choices that must be\n", "evaluated under realistic ACL selectivity. The safe contract preserves ACL\n", "metadata through parsing, chunking, candidate generation, reranking,\n", "generation, citations, logging, and caching.\n", "\n", "Deletion is likewise end to end. A source tombstone must invalidate every\n", "derived child chunk, embedding, postings entry, graph node/edge, summary,\n", "answer cache, training export, and replica. “Removed from the UI” is not an\n", "unlearning guarantee.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Update: updated version 2\n", "Tombstone: grounding-guide 2 source owner requested deletion\n", "Snapshot changed: True\n", "History versions: [1, 2]\n" ] } ], "source": [ "updated = manifest.ingest(\n", " Document(\n", " id=\"security-guide\",\n", " title=\"Security Guide\",\n", " source=\"https://example.test/security\",\n", " date=\"2026-08-04\",\n", " text=documents[2].text + \"\\n\\n## Audit\\nEvery release records immutable content hashes.\",\n", " ),\n", " acl=ACLPolicy.public(),\n", ")\n", "before_delete = manifest.snapshot_hash()\n", "tombstone = manifest.tombstone(\"grounding-guide\", \"source owner requested deletion\")\n", "print(\"Update:\", updated.action, \"version\", updated.entry.version)\n", "print(\"Tombstone:\", tombstone.document_id, tombstone.version, tombstone.tombstone_reason)\n", "print(\"Snapshot changed:\", before_delete != manifest.snapshot_hash())\n", "print(\"History versions:\", [entry.version for entry in manifest.history(\"grounding-guide\")])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Chunking defines the retrieval hypothesis space\n", "\n", "Common families include fixed token windows, sentences, paragraphs,\n", "Markdown/DOM sections, recursive separator splitting, discourse units,\n", "semantic-boundary segmentation, propositions, parent–child indexes,\n", "late chunking after long-document encoding, hierarchical summaries, table\n", "rows/regions, code symbols, graph nodes, and page-image patches.\n", "\n", "Smaller units improve localization and reduce distractors but lose context.\n", "Larger units preserve discourse but dilute similarity and consume the prompt.\n", "Overlap improves boundary recall while inflating storage, correlated\n", "candidates, and citation ambiguity. Tune chunking jointly with retriever,\n", "reranker, top-k, generator, and task; do not optimize a universal chunk size.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sentence chunks:\n", "retrieval-guide::sentence::0000 (0, 72) '# Sparse retrieval\\nAn inverted index stores postings for l'\n", "retrieval-guide::sentence::0001 (19, 72) 'An inverted index stores postings for lexical search.'\n", "retrieval-guide::sentence::0002 (73, 154) 'BM25 saturates term frequency and normalizes document leng'\n", "retrieval-guide::sentence::0003 (136, 209) '## Dense retrieval\\nA dual encoder maps queries and passage'\n", "retrieval-guide::sentence::0004 (155, 209) 'A dual encoder maps queries and passages into vectors.'\n", "retrieval-guide::sentence::0005 (210, 272) 'Approximate nearest-neighbor indexes trade recall for late'\n", "Section paths:\n", "retrieval-guide::section::0000 ('Sparse retrieval',) 18\n", "retrieval-guide::section::0001 ('Sparse retrieval', 'Dense retrieval') 18\n" ] } ], "source": [ "from rag_evolution.chunking import section_chunks, sentence_chunks\n", "\n", "sentence_view = sentence_chunks(documents[0], max_tokens=14, overlap_sentences=1)\n", "section_view = section_chunks(documents[0], max_tokens=35)\n", "print(\"Sentence chunks:\")\n", "for item in sentence_view:\n", " print(item.chunk.id, (item.lineage.start_char, item.lineage.end_char), repr(item.chunk.text[:58]))\n", "print(\"Section paths:\")\n", "for item in section_view:\n", " print(item.chunk.id, item.lineage.section_path, item.lineage.end_token - item.lineage.start_token)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Parent–child retrieval separates search granularity from reading granularity\n", "\n", "Retrieve a compact child because it has a sharp signal; send its larger\n", "parent because the generator needs definitions, qualifiers, or surrounding\n", "table rows. The child must store an exact parent edge and both need source\n", "coordinates. Parent expansion can otherwise silently exceed budgets or\n", "duplicate the same section several times.\n", "\n", "Evaluate child retrieval recall, parent expansion recall, packed evidence\n", "recall, duplicate rate, token cost, and citation precision separately.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Parents: [('retrieval-guide::parent::0000', ('Sparse retrieval',)), ('retrieval-guide::parent::0001', ('Sparse retrieval', 'Dense retrieval'))]\n", "Children -> parent:\n", "retrieval-guide::child::0000 -> retrieval-guide::parent::0000 chars (2, 72)\n", "retrieval-guide::child::0001 -> retrieval-guide::parent::0000 chars (73, 133)\n", "retrieval-guide::child::0002 -> retrieval-guide::parent::0001 chars (139, 209)\n", "retrieval-guide::child::0003 -> retrieval-guide::parent::0001 chars (210, 271)\n" ] } ], "source": [ "from rag_evolution.chunking import parent_child_chunks\n", "\n", "hierarchy = parent_child_chunks(\n", " documents[0], parent_max_tokens=38, child_max_tokens=14, child_overlap_sentences=0\n", ")\n", "print(\"Parents:\", [(item.chunk.id, item.lineage.section_path) for item in hierarchy.parents])\n", "print(\"Children -> parent:\")\n", "for item in hierarchy.children:\n", " print(item.chunk.id, \"->\", item.lineage.parent_chunk_id, \"chars\", (item.lineage.start_char, item.lineage.end_char))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Sparse retrieval is an execution engine\n", "\n", "An inverted index stores a postings list for each term. Query evaluation\n", "visits only postings for query terms, accumulates BM25/query-likelihood\n", "scores, and uses WAND/Block-Max WAND bounds to avoid fully scoring documents\n", "that cannot enter top-k. Fielded BM25, phrase/proximity, analyzers,\n", "stemming, multilingual tokenization, spelling, entity aliases, numeric/date\n", "handling, and pseudo-relevance feedback remain powerful—especially for\n", "identifiers, rare terms, code, names, and fresh vocabulary.\n", "\n", "Learned sparse models such as SPLADE retain inverted-index execution while\n", "learning expansion and term weights. Their operational questions include\n", "posting expansion, index size, latency, regularization, and domain drift.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Vocabulary terms: 86\n", "Postings for 'index': (Posting(document_ordinal=0, term_frequency=1),)\n", "Results: [('retrieval-guide', 7.03)]\n", "Pre-top-k ACL/filter result: ['security-guide']\n" ] } ], "source": [ "from rag_evolution.indexes import InvertedIndex\n", "\n", "index_chunks = tuple(item.chunk for doc in documents for item in sentence_chunks(doc, max_tokens=20, overlap_sentences=0))\n", "sparse_index = InvertedIndex(index_chunks)\n", "sparse_hits = sparse_index.search(\"inverted postings BM25 document length\", k=4)\n", "print(\"Vocabulary terms:\", len(sparse_index.postings))\n", "print(\"Postings for 'index':\", sparse_index.postings.get(\"index\", ()))\n", "print(\"Results:\", [(hit.chunk.document_id, round(hit.score, 3)) for hit in sparse_hits])\n", "allowed = {chunk.id for chunk in index_chunks if chunk.document_id == \"security-guide\"}\n", "print(\"Pre-top-k ACL/filter result:\", [hit.chunk.document_id for hit in sparse_index.search(\"index permissions\", 5, allowed)])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Dense retrieval needs an embedding contract\n", "\n", "Record model and tokenizer revision, pooling, normalization, distance\n", "metric, dimensionality, maximum input length, query/document prefixes,\n", "truncation, language/domain assumptions, batching precision, and training\n", "data. A mismatched prefix or cosine-vs-inner-product setting can invalidate\n", "an index without an obvious error.\n", "\n", "Exact search is the correctness oracle. Approximate search is a systems\n", "optimization and must be evaluated against exact neighbors on the current\n", "vector distribution. Retrieval quality additionally requires qrels: ANN\n", "recall only says whether the approximate engine reproduced exact embedding\n", "neighbors, not whether the embedding put relevant evidence nearby.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Vector dimensions: ('sparse', 'dense', 'evidence', 'security')\n", "Exact neighbors: [('retrieval-guide::sentence::0001', 1.0), ('retrieval-guide::sentence::0000', 0.196), ('grounding-guide::sentence::0000', 0.0), ('grounding-guide::sentence::0001', 0.0)]\n" ] } ], "source": [ "from rag_evolution.indexes import ExactCosineIndex\n", "from rag_evolution.text import tokenize\n", "\n", "axes = (\"sparse\", \"dense\", \"evidence\", \"security\")\n", "expansions = {\n", " \"sparse\": {\"sparse\", \"lexical\", \"bm25\", \"postings\", \"inverted\"},\n", " \"dense\": {\"dense\", \"vector\", \"encoder\", \"nearest-neighbor\"},\n", " \"evidence\": {\"evidence\", \"citation\", \"claim\", \"entailment\"},\n", " \"security\": {\"security\", \"permission\", \"tenant\", \"untrusted\", \"tombstone\"},\n", "}\n", "def lab_vector(text):\n", " terms = set(tokenize(text))\n", " return tuple(float(len(terms & expansions[axis])) for axis in axes)\n", "\n", "vectors = tuple(lab_vector(chunk.title + \" \" + chunk.text) for chunk in index_chunks)\n", "exact = ExactCosineIndex(index_chunks, vectors)\n", "query_vector = lab_vector(\"dense vector nearest-neighbor encoder\")\n", "exact_hits = exact.search(query_vector, k=4)\n", "print(\"Vector dimensions:\", axes)\n", "print(\"Exact neighbors:\", [(hit.chunk.id, round(hit.score, 3)) for hit in exact_hits])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. ANN families express different resource trade-offs\n", "\n", "- **HNSW** navigates a multilayer proximity graph; it offers strong recall and\n", " latency but consumes RAM and has update/filtering considerations.\n", "- **IVF** learns coarse cells and probes only likely lists; `nlist` and\n", " `nprobe` control build/search work and recall.\n", "- **PQ/OPQ** compress vectors into subspace codes; asymmetric distance tables\n", " trade reconstruction error for memory and bandwidth.\n", "- **DiskANN/SPANN** organize graph/centroid structures around SSD access for\n", " billion-scale collections.\n", "- **ScaNN** combines partitioning, quantization, and reordering.\n", "- Multi-vector indexes (ColBERT/ColPali) add token/patch-level storage and\n", " MaxSim execution; PLAID-style pruning reduces that cost.\n", "\n", "Measure recall@k versus exact search, qrels-based nDCG/recall, p50/p95\n", "latency, throughput, RAM/disk, build time, update/delete cost, filter\n", "selectivity, and performance under distribution shift.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "nprobe 1 mean ANN recall@3 0.667 per query (0.6666666666666666, 0.6666666666666666, 0.6666666666666666)\n", "nprobe 2 mean ANN recall@3 0.889 per query (1.0, 1.0, 0.6666666666666666)\n", "nprobe 3 mean ANN recall@3 1.0 per query (1.0, 1.0, 1.0)\n", "IVF lists: {0: (0, 1), 1: (2, 3), 2: (4, 5)}\n" ] } ], "source": [ "from rag_evolution.indexes import IVFCoarseIndex, evaluate_ann_recall\n", "\n", "ivf = IVFCoarseIndex(index_chunks, vectors, nlist=3, iterations=10)\n", "query_vectors = (\n", " lab_vector(\"dense vector encoder\"),\n", " lab_vector(\"citation evidence entailment\"),\n", " lab_vector(\"tenant permission security\"),\n", ")\n", "for nprobe in range(1, ivf.nlist + 1):\n", " audit = evaluate_ann_recall(exact, ivf, query_vectors, k=3, nprobe=nprobe)\n", " print(\"nprobe\", nprobe, \"mean ANN recall@3\", round(audit.mean_recall, 3), \"per query\", audit.per_query)\n", "print(\"IVF lists:\", dict(ivf.inverted_lists))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Compression must include codebooks and distortion\n", "\n", "Float16/8-bit scalar quantization, product quantization, binary codes,\n", "Matryoshka dimension truncation, pooling, and tiered hot/cold indexes reduce\n", "cost differently. Report code bytes *and* codebook/metadata bytes, build\n", "cost, reconstruction error, neighbor recall, downstream answer/citation\n", "quality, and hardware. Tiny toy corpora can have compression ratios below\n", "one because shared codebooks dominate; scale changes the accounting.\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "4-bit scalar {'original': 96, 'codes': 12, 'codebook': 32, 'ratio': 2.182, 'mse': 0.0002, 'cosine': 1.0}\n", "2x2-bit PQ {'original': 96, 'codes': 6, 'codebook': 64, 'ratio': 1.371, 'mse': 0.0278, 'cosine': 0.9974}\n" ] } ], "source": [ "from rag_evolution.indexes import ProductQuantizer, ScalarQuantizer, audit_quantization\n", "\n", "scalar = ScalarQuantizer(bits=4).fit(vectors)\n", "product = ProductQuantizer(subquantizers=2, bits=2, iterations=8).fit(vectors)\n", "for name, quantizer in ((\"4-bit scalar\", scalar), (\"2x2-bit PQ\", product)):\n", " audit = audit_quantization(vectors, quantizer)\n", " print(\n", " name,\n", " {\"original\": audit.original_bytes, \"codes\": audit.encoded_bytes,\n", " \"codebook\": audit.codebook_bytes, \"ratio\": round(audit.compression_ratio, 3),\n", " \"mse\": round(audit.mean_squared_error, 4),\n", " \"cosine\": round(audit.mean_cosine_similarity, 4)},\n", " )\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 10. Corpus/index release gate\n", "\n", "A releasable index records source snapshots, parser/chunker/embedding/index\n", "versions, permissions, exact-vs-ANN audit, duplicate policy, deletion replay,\n", "per-language/domain slices, storage, build/update latency, and rollback ID.\n", "Test boundary facts split across chunks, tables, captions, OCR corruption,\n", "rare identifiers, ACL-selective queries, changed/deleted sources, and\n", "adversarial duplicates. Compare chunkers under the same retriever and compare\n", "retrievers under the same chunks before claiming causality.\n", "\n", "**This lab does not reproduce** neural encoders, WAND, HNSW, DiskANN, or\n", "billion-scale performance. It supplies exact transparent baselines and the\n", "measurement contracts required to evaluate those implementations honestly.\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Release ID: fc5d49618ba0b200225296ad46096dd5973685cb65d00d82c6bb37a7a8299e9d\n", "Components: {'corpus_snapshot': 'abcfca0b7cdd09c1f18430388c6dd27a97c00c59b54007b5815fad02cb80c97b', 'parser_version': 'parser-lab-v1', 'chunker_version': 'lineage-chunker-v1', 'embedding_version': 'lab-vector-v1', 'index_version': 'ivf-v1', 'reranker_version': 'none', 'generator_version': 'none', 'prompt_version': 'none'}\n" ] } ], "source": [ "from rag_evolution.operations import release_manifest\n", "\n", "release = release_manifest(\n", " manifest.snapshot_hash(), \"parser-lab-v1\", \"lineage-chunker-v1\",\n", " \"lab-vector-v1\", \"ivf-v1\", \"none\", \"none\", \"none\"\n", ")\n", "print(\"Release ID:\", release[\"release_id\"])\n", "print(\"Components:\", {key: value for key, value in release.items() if key != \"release_id\"})\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.9" }, "rag_evolution": { "presentation": "plain-field-notebook", "visual_version": 2 } }, "nbformat": 4, "nbformat_minor": 5 }