{ "cells": [ { "cell_type": "markdown", "metadata": { "tags": [ "field-notebook-cover", "field-notebook-style" ] }, "source": [ "\n", "\n", "
\n", "
LAB 08 · OPERATIONS
\n", "

The System in the Weather

\n", "
Benchmarks, traces, SLOs, budgets, Pareto choices, releases, and recovery
\n", "
a system is not finished when the demo answers correctly
\n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The working page\n", "\n", "The final lab turns RAG into an operated system. It connects offline qrels,\n", "oracle-context generation, end-to-end answers, citations, abstention, safety,\n", "latency, cost, capacity, caching, drift, release identities, and rollback.\n", "\n", "**Learning outcomes**\n", "\n", "- evaluate retrieval, generation, citations, and end-to-end behavior separately;\n", "- retain per-query rows, slices, and paired uncertainty;\n", "- inspect a source-linked pipeline trace and abstention;\n", "- compute stage and service p50/p95/p99 metrics;\n", "- enforce hard request budgets and select a Pareto frontier;\n", "- create a content-addressed release manifest.\n", "\n", "Companion chapters: [Evaluation and risks](../research/evaluation_and_risks.md)\n", "and [Production systems](../research/production_systems.md).\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Repository root: resolved from the notebook location\n" ] } ], "source": [ "from pathlib import Path\n", "import sys\n", "\n", "ROOT = Path.cwd()\n", "if not (ROOT / \"src\").exists():\n", " ROOT = ROOT.parent\n", "sys.path.insert(0, str(ROOT / \"src\"))\n", "print(\"Repository root: resolved from the notebook location\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Separate control, data, serving, and evaluation planes\n", "\n", "The **data plane** connects sources, parses, versions, deduplicates, chunks,\n", "embeds, indexes, applies ACL metadata, and propagates deletion. The **serving\n", "plane** authenticates, classifies/routes, retrieves, fuses, reranks, selects,\n", "generates, verifies, cites, and logs. The **control plane** versions configs,\n", "models, prompts, schemas, releases, rollouts, budgets, and policies. The\n", "**evaluation plane** owns qrels, gold claims, adversarial suites, judges,\n", "human audits, regression gates, and experiment traces.\n", "\n", "A request should carry caller/tenant, query time, corpus/index release,\n", "retriever/reranker/generator/prompt versions, retrieved IDs/scores, packed\n", "spans, answer claims/citations, decisions, tokens, timings, cost, cache status,\n", "and errors—with sensitive content minimized or redacted.\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks coupled a DPR question encoder with a frozen passage index and a BART generator. [rag-2020::c000] Dense Passage Retrieval (DPR) for Open-Domain Question Answering introduced a simple dual-encoder retriever trained with positive passages, in-batch negatives, and a hard BM25 negative. [dpr-2020::c000]\n", "Citations: [('rag-2020', 'https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html'), ('dpr-2020', 'https://aclanthology.org/2020.emnlp-main.550/')]\n", "Trace:\n", "route selected graph retrieval {'route': 'graph'}\n", "retrieve retrieved 14 candidates {'count': 14, 'k': 14}\n", "rerank retained 8 reranked candidates {'count': 8, 'k': 8}\n", "pack packed 6 non-redundant chunks {'count': 6, 'max_tokens': 560}\n", "generate returned grounded evidence {'citations': 2, 'confidence': 0.61, 'abstained': False}\n" ] } ], "source": [ "from rag_evolution.demo_data import demo_documents, demo_questions\n", "from rag_evolution.pipeline import build_advanced_pipeline, build_baseline_pipeline\n", "\n", "documents = demo_documents()\n", "questions = demo_questions()\n", "baseline = build_baseline_pipeline(documents)\n", "advanced = build_advanced_pipeline(documents)\n", "answer = advanced.ask(\"What is the relationship and difference between DPR and the original RAG model?\")\n", "print(answer.text)\n", "print(\"Citations:\", [(citation.document_id, citation.source) for citation in answer.citations])\n", "print(\"Trace:\")\n", "for event in answer.trace:\n", " print(event.stage, event.detail, dict(event.values))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Evaluation is a stack, not one “RAG score”\n", "\n", "**Retrieval:** Recall@k, precision@k, MRR, MAP, nDCG, context/claim recall,\n", "first supporting rank, duplicate rate, temporal/authority correctness, ANN\n", "recall, latency. **Oracle-context generation:** claim precision/recall,\n", "correctness, completeness, faithfulness, context utilization, citation\n", "entailment/completeness, abstention/calibration. **End to end:** all of those\n", "plus failure attribution, task utility, safety, cost, and latency.\n", "\n", "RAGAS/ARES/RAGChecker are evaluator frameworks, not interchangeable task\n", "leaderboards. BEIR/MTEB/BRIGHT evaluate retrieval. KILT/TREC RAG emphasize\n", "provenance/citations. CRAG/RGB/CRUD-RAG/mtRAG stress freshness, noise,\n", "lifecycle, or conversation. Select benchmarks by product risk and maintain a\n", "stratified product gold set.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Baseline retrieval: {'precision@5': 0.225, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 0.985}\n", "Advanced retrieval: {'precision@5': 0.225, 'recall@5': 1.0, 'mrr': 0.938, 'ndcg@5': 0.954}\n", "Per-query advanced rows:\n", "{'id': 'q-dpr', 'tags': ('single-hop', 'lexical'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}\n", "{'id': 'q-fid', 'tags': ('single-hop', 'architecture'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}\n", "{'id': 'q-rag-dpr', 'tags': ('multi-hop', 'comparison'), 'precision@5': 0.4, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}\n", "{'id': 'q-self-rag', 'tags': ('single-hop', 'adaptive'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}\n", "{'id': 'q-raptor', 'tags': ('single-hop', 'hierarchical'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 0.5, 'ndcg@5': 0.6309297535714575}\n", "{'id': 'q-visual', 'tags': ('single-hop', 'multimodal'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}\n", "{'id': 'q-long-context', 'tags': ('single-hop', 'routing'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}\n", "{'id': 'q-grip', 'tags': ('single-hop', 'agentic'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}\n" ] } ], "source": [ "from rag_evolution.evaluation import aggregate_metrics, evaluate_retriever\n", "\n", "baseline_rows = evaluate_retriever(baseline, questions, k=5)\n", "advanced_rows = evaluate_retriever(advanced, questions, k=5)\n", "print(\"Baseline retrieval:\", {key: round(value, 3) for key, value in aggregate_metrics(baseline_rows).items()})\n", "print(\"Advanced retrieval:\", {key: round(value, 3) for key, value in aggregate_metrics(advanced_rows).items()})\n", "print(\"Per-query advanced rows:\")\n", "for row in advanced_rows:\n", " print(row)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Keep oracle and closed-book controls\n", "\n", "Closed-book generation measures parametric knowledge. Oracle-context\n", "generation measures whether the reader/generator can use perfect evidence.\n", "Retrieved-context generation adds retrieval and packing. Distractor controls\n", "test reader robustness. Citation-removed and source-shuffled controls reveal\n", "whether apparent grounding comes from evidence. An answer-only metric can\n", "reward unsupported model knowledge; strict context-only faithfulness can\n", "penalize true but uncited facts. Report both policy and metric semantics.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Baseline end-to-end: {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.275, 'citation_precision': 0.875, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.678}\n", "Advanced end-to-end: {'recall@5': 1.0, 'mrr': 0.938, 'answer_em': 0.0, 'answer_f1': 0.275, 'citation_precision': 0.875, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.678}\n" ] } ], "source": [ "from rag_evolution.evaluation import evaluate_pipeline\n", "\n", "baseline_answers = evaluate_pipeline(baseline, questions)\n", "advanced_answers = evaluate_pipeline(advanced, questions)\n", "print(\"Baseline end-to-end:\", {key: round(value, 3) for key, value in aggregate_metrics(baseline_answers).items()})\n", "print(\"Advanced end-to-end:\", {key: round(value, 3) for key, value in aggregate_metrics(advanced_answers).items()})\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Slices and uncertainty prevent average-score theater\n", "\n", "Stratify answerable/partial/unanswerable, long-tail/popular, fresh/historical,\n", "single/multi-hop, conflicting/noisy, language, modality, table/long document,\n", "conversation turn, tenant/ACL selectivity, and safety attack. Preserve\n", "per-query paired results; bootstrap confidence intervals or use calibrated\n", "aggregate estimators such as ARES prediction-powered inference. Double-label\n", "and adjudicate a human slice; report judge prompts/models and agreement.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Advanced metrics by tag:\n", "adaptive {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.424, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.856}\n", "agentic {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.049, 'citation_precision': 0.5, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.536}\n", "architecture {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.391, 'citation_precision': 0.5, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.621}\n", "comparison {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.286, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.61}\n", "hierarchical {'recall@5': 1.0, 'mrr': 0.5, 'answer_em': 0.0, 'answer_f1': 0.323, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.675}\n", "lexical {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.485, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.594}\n", "multi-hop {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.286, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.61}\n", "multimodal {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.08, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.814}\n", "routing {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.162, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.721}\n", "single-hop {'recall@5': 1.0, 'mrr': 0.929, 'answer_em': 0.0, 'answer_f1': 0.273, 'citation_precision': 0.857, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.688}\n", "Paired answer-F1 delta and 95% interval: (0.0, 0.0, 0.0)\n" ] } ], "source": [ "from rag_evolution.evaluation import metrics_by_tag, paired_bootstrap_delta\n", "\n", "print(\"Advanced metrics by tag:\")\n", "for tag, metrics in metrics_by_tag(advanced_answers).items():\n", " print(tag, {key: round(value, 3) for key, value in metrics.items()})\n", "baseline_f1 = [row[\"answer_f1\"] for row in baseline_answers]\n", "advanced_f1 = [row[\"answer_f1\"] for row in advanced_answers]\n", "delta, lower, upper = paired_bootstrap_delta(advanced_f1, baseline_f1, iterations=1000, seed=17)\n", "print(\"Paired answer-F1 delta and 95% interval:\", tuple(round(value, 4) for value in (delta, lower, upper)))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Failure attribution follows evidence survival\n", "\n", "For each failed claim, ask: source absent from corpus; parser corrupted it;\n", "chunk boundary split it; embedding/sparse candidate missed it; ANN missed the\n", "exact neighbor; query transformation changed intent; fusion/reranker removed\n", "it; packer dropped it; generator ignored/misread it; verifier failed; citation\n", "mapped to the wrong span; stale/unauthorized cache intervened. This taxonomy\n", "turns a vague “RAG failed” into an owned component regression.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "EvidenceFlow(relevant=('dpr-2020', 'rag-2020'), retrieved=('rag-2020', 'dpr-2020', 'search-r1-2025', 'lara-2025', 'atlas-2022', 'self-rag-2023', 'fid-2021', 'grip-2026', 'colpali-2024', 'raptor-2024'), reranked=('rag-2020', 'dpr-2020', 'search-r1-2025', 'lara-2025', 'atlas-2022', 'self-rag-2023'), packed=('rag-2020', 'dpr-2020', 'search-r1-2025'), retrieval_recall=1.0, rerank_survival=1.0, pack_survival=1.0, end_to_end_recall=1.0, lost_at_retrieval=(), lost_at_rerank=(), lost_at_pack=())\n" ] } ], "source": [ "from rag_evolution.context import ContextPacker\n", "from rag_evolution.selection import evidence_flow\n", "\n", "example = next(item for item in questions if item.id == \"q-rag-dpr\")\n", "candidates = advanced.search(example.question, 10)\n", "reranked = candidates[:6]\n", "packed = ContextPacker(max_tokens=220, max_chunks=3).pack(reranked)\n", "flow = evidence_flow(example.relevant_document_ids, candidates, reranked, packed)\n", "print(flow)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Operational SLOs are stage-specific\n", "\n", "Track p50/p95/p99 for authentication/routing, sparse/dense retrieval, fusion,\n", "reranking, evidence fetch, packing, model time-to-first-token, generation,\n", "verification, and end-to-end. Also index bytes/document, ingest/update lag,\n", "throughput, queue time, calls, prompt/completion tokens, cache hit, cost per\n", "successful cited answer, error/timeout/degraded-mode rates, citation/support,\n", "stale-answer rate, and security events.\n", "\n", "Tail latency matters: agentic steps are often sequential and multiply\n", "variance. Enforce deadlines and cancellation; batch embeddings/reranking;\n", "use tiered indexes/caches; and define degraded paths such as sparse-only,\n", "no-reranker, smaller model, or abstention.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Service: {'p50_latency_ms': 219.0, 'p95_latency_ms': 419.65, 'p99_latency_ms': 439.93, 'mean_cost_usd': 0.0253, 'failure_rate': 0.0, 'safe_rate': 1.0, 'citation_rate': 1.0, 'abstention_rate': 0.0, 'cache_hit_rate': 0.0833}\n", "Stages:\n", "generate {'count': 4.0, 'p50_ms': 165.0, 'p95_ms': 312.0, 'mean_cost_usd': 0.0253, 'mean_calls': 1.0, 'error_rate': 0.0}\n", "rerank {'count': 4.0, 'p50_ms': 27.5, 'p95_ms': 42.9, 'mean_cost_usd': 0.0, 'mean_calls': 1.0, 'error_rate': 0.0}\n", "retrieve {'count': 4.0, 'p50_ms': 26.5, 'p95_ms': 64.75, 'mean_cost_usd': 0.0, 'mean_calls': 1.75, 'error_rate': 0.0}\n" ] } ], "source": [ "from rag_evolution.operations import RequestMeasurement, StageMeasurement, latency_summary, stage_summary\n", "\n", "requests = (\n", " RequestMeasurement(\"r1\", (StageMeasurement(\"retrieve\", 18, calls=1, cache_hit=True), StageMeasurement(\"rerank\", 24), StageMeasurement(\"generate\", 120, 0.018, 420, 110)), 0.84, True, False),\n", " RequestMeasurement(\"r2\", (StageMeasurement(\"retrieve\", 35, calls=2), StageMeasurement(\"rerank\", 31), StageMeasurement(\"generate\", 210, 0.029, 710, 180)), 0.91, True, False),\n", " RequestMeasurement(\"r3\", (StageMeasurement(\"retrieve\", 15, calls=1), StageMeasurement(\"rerank\", 20), StageMeasurement(\"generate\", 95, 0.013, 300, 80)), 0.72, True, False),\n", " RequestMeasurement(\"r4\", (StageMeasurement(\"retrieve\", 70, calls=3), StageMeasurement(\"rerank\", 45), StageMeasurement(\"generate\", 330, 0.041, 980, 250)), 0.93, True, False),\n", ")\n", "print(\"Service:\", {key: round(value, 4) for key, value in latency_summary(requests).items()})\n", "print(\"Stages:\")\n", "for stage, metrics in stage_summary(requests).items():\n", " print(stage, {key: round(value, 4) for key, value in metrics.items()})\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Hard budgets remain outside the model\n", "\n", "An agent cannot be trusted to enforce its own maximum spend, calls, output\n", "tokens, tool scopes, or deadline. Infrastructure must reject or cancel\n", "actions beyond the budget and log the reason. Budget violations and quality\n", "under degraded mode belong in release tests.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "r1 allowed True violations ()\n", "r2 allowed True violations ()\n", "r3 allowed True violations ()\n", "r4 allowed False violations ('latency', 'cost', 'retrieval_calls', 'generation_tokens')\n" ] } ], "source": [ "from rag_evolution.operations import ServiceBudget, check_budget\n", "\n", "budget = ServiceBudget(\n", " maximum_latency_ms=300,\n", " maximum_cost_usd=0.03,\n", " maximum_retrieval_calls=2,\n", " maximum_generation_tokens=200,\n", ")\n", "for request in requests:\n", " check = check_budget(request, budget)\n", " print(request.request_id, \"allowed\", check.allowed, \"violations\", check.violations)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. Optimize a constrained utility, not accuracy alone\n", "\n", "A useful framing is\n", "\n", "\\[\n", "U = Q - \\lambda_c C - \\lambda_l L - \\lambda_r R,\n", "\\]\n", "\n", "subject to hard safety, privacy, correctness, and latency gates. A Pareto\n", "frontier contains configurations not dominated simultaneously on quality,\n", "cost, latency, and risk. Choose weights only after plotting the frontier and\n", "checking product constraints; a single average can hide catastrophic slices.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Pareto frontier: ['agentic', 'hybrid-reranked', 'sparse']\n", "Release-feasible: ['hybrid-reranked']\n", "sparse utility 0.563\n", "hybrid-reranked utility 0.663\n", "agentic utility 0.45\n", "worse-copy utility 0.475\n" ] } ], "source": [ "from rag_evolution.operations import SystemCandidate, constrained_choice, pareto_frontier, utility\n", "\n", "systems = (\n", " SystemCandidate(\"sparse\", 0.68, 0.004, 90, 0.08),\n", " SystemCandidate(\"hybrid-reranked\", 0.84, 0.018, 170, 0.07),\n", " SystemCandidate(\"agentic\", 0.88, 0.052, 460, 0.12),\n", " SystemCandidate(\"worse-copy\", 0.64, 0.010, 130, 0.10),\n", ")\n", "print(\"Pareto frontier:\", [item.name for item in pareto_frontier(systems)])\n", "print(\"Release-feasible:\", [item.name for item in constrained_choice(systems, 0.75, 0.03, 250, 0.10)])\n", "for item in systems:\n", " print(item.name, \"utility\", round(utility(item, cost_weight=2.0, latency_weight=0.0005, risk_weight=0.8), 4))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Reproducibility requires a complete release identity\n", "\n", "Pin corpus snapshot/query time, source hashes and permissions, parser,\n", "chunker/overlap, embedding/prefix, sparse analyzer, ANN parameters,\n", "retrievers/fusion/reranker, top-k, context selector/budget/order, model,\n", "prompt/schema, decoding/seed, caches, judge, qrels, hardware, and code commit.\n", "Store per-query retrieved text/scores, decisions, output claims/citations,\n", "latency, cost, and errors. A model family name without revision is not a\n", "reproducible configuration.\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Release: d6aa3d97e5675f67a2b1d0413b222e2f255bc97edd62b7b3f40ca8c7b963b4ae\n", "Experiment: 46af201c8bbf0b27d586c9d0b305bf6a3ba9a830094b3def684d8dd3cc79bfbb\n" ] } ], "source": [ "from rag_evolution.operations import configuration_fingerprint, release_manifest\n", "\n", "release = release_manifest(\n", " \"corpus-sha-91\", \"docling-2.4\", \"section-child-3\", \"embed-r17\",\n", " \"hnsw-m32-ef200\", \"crossencoder-r8\", \"generator-r12\", \"prompt-r31\"\n", ")\n", "experiment = {\n", " **release,\n", " \"fusion\": {\"method\": \"rrf\", \"constant\": 60},\n", " \"candidate_k\": 80,\n", " \"rerank_k\": 12,\n", " \"context_tokens\": 6000,\n", " \"qrels\": \"product-gold-2026-08\",\n", " \"seed\": 17,\n", "}\n", "print(\"Release:\", release[\"release_id\"])\n", "print(\"Experiment:\", configuration_fingerprint(experiment))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 10. Release, rollout, monitoring, and rollback\n", "\n", "Offline gates: deterministic unit/golden tests; retrieval/citation/safety\n", "thresholds; deletion/ACL/freshness tests; adversarial corpus tests; paired\n", "confidence; latency/cost/capacity bounds. Online: shadow, canary by tenant,\n", "A/B or interleaving where valid, kill switch, rollback compatible with index\n", "schema. Monitor data/parser/chunk/embedding/query/score/route/output drift,\n", "support/citation/abstention, incidents, SLOs, and business outcomes.\n", "\n", "Maintain runbooks for source outage, stale index, ANN corruption, model/API\n", "outage, cache poisoning, cross-tenant leak, deletion failure, cost runaway,\n", "prompt injection, and bad rollout. Practice restore and replay.\n", "\n", "**This notebook is the operational acceptance test, not a leaderboard.** A\n", "system ships only when its task slices, evidence guarantees, security gates,\n", "and resource envelope meet the product contract.\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "tags": [ "binding-instrument", "publication-instrument" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Notebook artifacts: ['00_complete_rag_handbook.ipynb', '01_rag_evolution.ipynb', '02_advanced_rag.ipynb', '03_evaluation_and_failure_analysis.ipynb', '04_corpus_chunking_and_indexes.ipynb', '05_training_query_fusion_and_reranking.ipynb', '06_structured_multimodal_and_graph_rag.ipynb', '07_agents_memory_temporal_and_security.ipynb', '08_production_evaluation_and_cost.ipynb']\n", "Reference modules: ['__init__', 'agentic', 'chunking', 'context', 'demo_data', 'evaluation', 'generation', 'indexes', 'ingestion', 'memory', 'models', 'operations', 'pipeline', 'rerankers', 'retrievers', 'security', 'selection', 'structured', 'temporal', 'text', 'training']\n", "Discovered test methods: 129\n", "Handbook Markdown files: 18\n" ] } ], "source": [ "import ast\n", "import json\n", "import re\n", "\n", "notebook_files = sorted((ROOT / \"notebooks\").glob(\"*.ipynb\"))\n", "module_files = sorted((ROOT / \"src\" / \"rag_evolution\").glob(\"*.py\"))\n", "test_files = sorted((ROOT / \"tests\").glob(\"test_*.py\"))\n", "test_count = 0\n", "for path in test_files:\n", " tree = ast.parse(path.read_text(encoding=\"utf-8\"))\n", " test_count += sum(\n", " isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name.startswith(\"test_\")\n", " for node in ast.walk(tree)\n", " )\n", "print(\"Notebook artifacts:\", [path.name for path in notebook_files])\n", "print(\"Reference modules:\", [path.stem for path in module_files])\n", "print(\"Discovered test methods:\", test_count)\n", "print(\"Handbook Markdown files:\", len(list((ROOT / \"research\").glob(\"*.md\"))))\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.9" }, "rag_evolution": { "presentation": "plain-field-notebook", "visual_version": 2 } }, "nbformat": 4, "nbformat_minor": 5 }