The System in the Weather
The working page
The final lab turns RAG into an operated system. It connects offline qrels, oracle-context generation, end-to-end answers, citations, abstention, safety, latency, cost, capacity, caching, drift, release identities, and rollback.
Learning outcomes
- evaluate retrieval, generation, citations, and end-to-end behavior separately;
- retain per-query rows, slices, and paired uncertainty;
- inspect a source-linked pipeline trace and abstention;
- compute stage and service p50/p95/p99 metrics;
- enforce hard request budgets and select a Pareto frontier;
- create a content-addressed release manifest.
Companion chapters: Evaluation and risks and Production systems.
from pathlib import Path
import sys
ROOT = Path.cwd()
if not (ROOT / "src").exists():
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / "src"))
print("Repository root: resolved from the notebook location")
Repository root: resolved from the notebook location
1. Separate control, data, serving, and evaluation planes
The data plane connects sources, parses, versions, deduplicates, chunks, embeds, indexes, applies ACL metadata, and propagates deletion. The serving plane authenticates, classifies/routes, retrieves, fuses, reranks, selects, generates, verifies, cites, and logs. The control plane versions configs, models, prompts, schemas, releases, rollouts, budgets, and policies. The evaluation plane owns qrels, gold claims, adversarial suites, judges, human audits, regression gates, and experiment traces.
A request should carry caller/tenant, query time, corpus/index release, retriever/reranker/generator/prompt versions, retrieved IDs/scores, packed spans, answer claims/citations, decisions, tokens, timings, cost, cache status, and errors—with sensitive content minimized or redacted.
from rag_evolution.demo_data import demo_documents, demo_questions
from rag_evolution.pipeline import build_advanced_pipeline, build_baseline_pipeline
documents = demo_documents()
questions = demo_questions()
baseline = build_baseline_pipeline(documents)
advanced = build_advanced_pipeline(documents)
answer = advanced.ask("What is the relationship and difference between DPR and the original RAG model?")
print(answer.text)
print("Citations:", [(citation.document_id, citation.source) for citation in answer.citations])
print("Trace:")
for event in answer.trace:
print(event.stage, event.detail, dict(event.values))
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks coupled a DPR question encoder with a frozen passage index and a BART generator. [rag-2020::c000] Dense Passage Retrieval (DPR) for Open-Domain Question Answering introduced a simple dual-encoder retriever trained with positive passages, in-batch negatives, and a hard BM25 negative. [dpr-2020::c000]
Citations: [('rag-2020', 'https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html'), ('dpr-2020', 'https://aclanthology.org/2020.emnlp-main.550/')]
Trace:
route selected graph retrieval {'route': 'graph'}
retrieve retrieved 14 candidates {'count': 14, 'k': 14}
rerank retained 8 reranked candidates {'count': 8, 'k': 8}
pack packed 6 non-redundant chunks {'count': 6, 'max_tokens': 560}
generate returned grounded evidence {'citations': 2, 'confidence': 0.61, 'abstained': False}
2. Evaluation is a stack, not one “RAG score”
Retrieval: Recall@k, precision@k, MRR, MAP, nDCG, context/claim recall, first supporting rank, duplicate rate, temporal/authority correctness, ANN recall, latency. Oracle-context generation: claim precision/recall, correctness, completeness, faithfulness, context utilization, citation entailment/completeness, abstention/calibration. End to end: all of those plus failure attribution, task utility, safety, cost, and latency.
RAGAS/ARES/RAGChecker are evaluator frameworks, not interchangeable task leaderboards. BEIR/MTEB/BRIGHT evaluate retrieval. KILT/TREC RAG emphasize provenance/citations. CRAG/RGB/CRUD-RAG/mtRAG stress freshness, noise, lifecycle, or conversation. Select benchmarks by product risk and maintain a stratified product gold set.
from rag_evolution.evaluation import aggregate_metrics, evaluate_retriever
baseline_rows = evaluate_retriever(baseline, questions, k=5)
advanced_rows = evaluate_retriever(advanced, questions, k=5)
print("Baseline retrieval:", {key: round(value, 3) for key, value in aggregate_metrics(baseline_rows).items()})
print("Advanced retrieval:", {key: round(value, 3) for key, value in aggregate_metrics(advanced_rows).items()})
print("Per-query advanced rows:")
for row in advanced_rows:
print(row)
Baseline retrieval: {'precision@5': 0.225, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 0.985}
Advanced retrieval: {'precision@5': 0.225, 'recall@5': 1.0, 'mrr': 0.938, 'ndcg@5': 0.954}
Per-query advanced rows:
{'id': 'q-dpr', 'tags': ('single-hop', 'lexical'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}
{'id': 'q-fid', 'tags': ('single-hop', 'architecture'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}
{'id': 'q-rag-dpr', 'tags': ('multi-hop', 'comparison'), 'precision@5': 0.4, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}
{'id': 'q-self-rag', 'tags': ('single-hop', 'adaptive'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}
{'id': 'q-raptor', 'tags': ('single-hop', 'hierarchical'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 0.5, 'ndcg@5': 0.6309297535714575}
{'id': 'q-visual', 'tags': ('single-hop', 'multimodal'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}
{'id': 'q-long-context', 'tags': ('single-hop', 'routing'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}
{'id': 'q-grip', 'tags': ('single-hop', 'agentic'), 'precision@5': 0.2, 'recall@5': 1.0, 'mrr': 1.0, 'ndcg@5': 1.0}
3. Keep oracle and closed-book controls
Closed-book generation measures parametric knowledge. Oracle-context generation measures whether the reader/generator can use perfect evidence. Retrieved-context generation adds retrieval and packing. Distractor controls test reader robustness. Citation-removed and source-shuffled controls reveal whether apparent grounding comes from evidence. An answer-only metric can reward unsupported model knowledge; strict context-only faithfulness can penalize true but uncited facts. Report both policy and metric semantics.
from rag_evolution.evaluation import evaluate_pipeline
baseline_answers = evaluate_pipeline(baseline, questions)
advanced_answers = evaluate_pipeline(advanced, questions)
print("Baseline end-to-end:", {key: round(value, 3) for key, value in aggregate_metrics(baseline_answers).items()})
print("Advanced end-to-end:", {key: round(value, 3) for key, value in aggregate_metrics(advanced_answers).items()})
Baseline end-to-end: {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.275, 'citation_precision': 0.875, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.678}
Advanced end-to-end: {'recall@5': 1.0, 'mrr': 0.938, 'answer_em': 0.0, 'answer_f1': 0.275, 'citation_precision': 0.875, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.678}
4. Slices and uncertainty prevent average-score theater
Stratify answerable/partial/unanswerable, long-tail/popular, fresh/historical, single/multi-hop, conflicting/noisy, language, modality, table/long document, conversation turn, tenant/ACL selectivity, and safety attack. Preserve per-query paired results; bootstrap confidence intervals or use calibrated aggregate estimators such as ARES prediction-powered inference. Double-label and adjudicate a human slice; report judge prompts/models and agreement.
from rag_evolution.evaluation import metrics_by_tag, paired_bootstrap_delta
print("Advanced metrics by tag:")
for tag, metrics in metrics_by_tag(advanced_answers).items():
print(tag, {key: round(value, 3) for key, value in metrics.items()})
baseline_f1 = [row["answer_f1"] for row in baseline_answers]
advanced_f1 = [row["answer_f1"] for row in advanced_answers]
delta, lower, upper = paired_bootstrap_delta(advanced_f1, baseline_f1, iterations=1000, seed=17)
print("Paired answer-F1 delta and 95% interval:", tuple(round(value, 4) for value in (delta, lower, upper)))
Advanced metrics by tag:
adaptive {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.424, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.856}
agentic {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.049, 'citation_precision': 0.5, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.536}
architecture {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.391, 'citation_precision': 0.5, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.621}
comparison {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.286, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.61}
hierarchical {'recall@5': 1.0, 'mrr': 0.5, 'answer_em': 0.0, 'answer_f1': 0.323, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.675}
lexical {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.485, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.594}
multi-hop {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.286, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.61}
multimodal {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.08, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.814}
routing {'recall@5': 1.0, 'mrr': 1.0, 'answer_em': 0.0, 'answer_f1': 0.162, 'citation_precision': 1.0, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.721}
single-hop {'recall@5': 1.0, 'mrr': 0.929, 'answer_em': 0.0, 'answer_f1': 0.273, 'citation_precision': 0.857, 'citation_recall': 1.0, 'lexical_faithfulness': 1.0, 'abstained': 0.0, 'confidence': 0.688}
Paired answer-F1 delta and 95% interval: (0.0, 0.0, 0.0)
5. Failure attribution follows evidence survival
For each failed claim, ask: source absent from corpus; parser corrupted it; chunk boundary split it; embedding/sparse candidate missed it; ANN missed the exact neighbor; query transformation changed intent; fusion/reranker removed it; packer dropped it; generator ignored/misread it; verifier failed; citation mapped to the wrong span; stale/unauthorized cache intervened. This taxonomy turns a vague “RAG failed” into an owned component regression.
from rag_evolution.context import ContextPacker
from rag_evolution.selection import evidence_flow
example = next(item for item in questions if item.id == "q-rag-dpr")
candidates = advanced.search(example.question, 10)
reranked = candidates[:6]
packed = ContextPacker(max_tokens=220, max_chunks=3).pack(reranked)
flow = evidence_flow(example.relevant_document_ids, candidates, reranked, packed)
print(flow)
EvidenceFlow(relevant=('dpr-2020', 'rag-2020'), retrieved=('rag-2020', 'dpr-2020', 'search-r1-2025', 'lara-2025', 'atlas-2022', 'self-rag-2023', 'fid-2021', 'grip-2026', 'colpali-2024', 'raptor-2024'), reranked=('rag-2020', 'dpr-2020', 'search-r1-2025', 'lara-2025', 'atlas-2022', 'self-rag-2023'), packed=('rag-2020', 'dpr-2020', 'search-r1-2025'), retrieval_recall=1.0, rerank_survival=1.0, pack_survival=1.0, end_to_end_recall=1.0, lost_at_retrieval=(), lost_at_rerank=(), lost_at_pack=())
6. Operational SLOs are stage-specific
Track p50/p95/p99 for authentication/routing, sparse/dense retrieval, fusion, reranking, evidence fetch, packing, model time-to-first-token, generation, verification, and end-to-end. Also index bytes/document, ingest/update lag, throughput, queue time, calls, prompt/completion tokens, cache hit, cost per successful cited answer, error/timeout/degraded-mode rates, citation/support, stale-answer rate, and security events.
Tail latency matters: agentic steps are often sequential and multiply variance. Enforce deadlines and cancellation; batch embeddings/reranking; use tiered indexes/caches; and define degraded paths such as sparse-only, no-reranker, smaller model, or abstention.
from rag_evolution.operations import RequestMeasurement, StageMeasurement, latency_summary, stage_summary
requests = (
RequestMeasurement("r1", (StageMeasurement("retrieve", 18, calls=1, cache_hit=True), StageMeasurement("rerank", 24), StageMeasurement("generate", 120, 0.018, 420, 110)), 0.84, True, False),
RequestMeasurement("r2", (StageMeasurement("retrieve", 35, calls=2), StageMeasurement("rerank", 31), StageMeasurement("generate", 210, 0.029, 710, 180)), 0.91, True, False),
RequestMeasurement("r3", (StageMeasurement("retrieve", 15, calls=1), StageMeasurement("rerank", 20), StageMeasurement("generate", 95, 0.013, 300, 80)), 0.72, True, False),
RequestMeasurement("r4", (StageMeasurement("retrieve", 70, calls=3), StageMeasurement("rerank", 45), StageMeasurement("generate", 330, 0.041, 980, 250)), 0.93, True, False),
)
print("Service:", {key: round(value, 4) for key, value in latency_summary(requests).items()})
print("Stages:")
for stage, metrics in stage_summary(requests).items():
print(stage, {key: round(value, 4) for key, value in metrics.items()})
Service: {'p50_latency_ms': 219.0, 'p95_latency_ms': 419.65, 'p99_latency_ms': 439.93, 'mean_cost_usd': 0.0253, 'failure_rate': 0.0, 'safe_rate': 1.0, 'citation_rate': 1.0, 'abstention_rate': 0.0, 'cache_hit_rate': 0.0833}
Stages:
generate {'count': 4.0, 'p50_ms': 165.0, 'p95_ms': 312.0, 'mean_cost_usd': 0.0253, 'mean_calls': 1.0, 'error_rate': 0.0}
rerank {'count': 4.0, 'p50_ms': 27.5, 'p95_ms': 42.9, 'mean_cost_usd': 0.0, 'mean_calls': 1.0, 'error_rate': 0.0}
retrieve {'count': 4.0, 'p50_ms': 26.5, 'p95_ms': 64.75, 'mean_cost_usd': 0.0, 'mean_calls': 1.75, 'error_rate': 0.0}
7. Hard budgets remain outside the model
An agent cannot be trusted to enforce its own maximum spend, calls, output tokens, tool scopes, or deadline. Infrastructure must reject or cancel actions beyond the budget and log the reason. Budget violations and quality under degraded mode belong in release tests.
from rag_evolution.operations import ServiceBudget, check_budget
budget = ServiceBudget(
maximum_latency_ms=300,
maximum_cost_usd=0.03,
maximum_retrieval_calls=2,
maximum_generation_tokens=200,
)
for request in requests:
check = check_budget(request, budget)
print(request.request_id, "allowed", check.allowed, "violations", check.violations)
r1 allowed True violations ()
r2 allowed True violations ()
r3 allowed True violations ()
r4 allowed False violations ('latency', 'cost', 'retrieval_calls', 'generation_tokens')
8. Optimize a constrained utility, not accuracy alone
A useful framing is
subject to hard safety, privacy, correctness, and latency gates. A Pareto frontier contains configurations not dominated simultaneously on quality, cost, latency, and risk. Choose weights only after plotting the frontier and checking product constraints; a single average can hide catastrophic slices.
from rag_evolution.operations import SystemCandidate, constrained_choice, pareto_frontier, utility
systems = (
SystemCandidate("sparse", 0.68, 0.004, 90, 0.08),
SystemCandidate("hybrid-reranked", 0.84, 0.018, 170, 0.07),
SystemCandidate("agentic", 0.88, 0.052, 460, 0.12),
SystemCandidate("worse-copy", 0.64, 0.010, 130, 0.10),
)
print("Pareto frontier:", [item.name for item in pareto_frontier(systems)])
print("Release-feasible:", [item.name for item in constrained_choice(systems, 0.75, 0.03, 250, 0.10)])
for item in systems:
print(item.name, "utility", round(utility(item, cost_weight=2.0, latency_weight=0.0005, risk_weight=0.8), 4))
Pareto frontier: ['agentic', 'hybrid-reranked', 'sparse'] Release-feasible: ['hybrid-reranked'] sparse utility 0.563 hybrid-reranked utility 0.663 agentic utility 0.45 worse-copy utility 0.475
9. Reproducibility requires a complete release identity
Pin corpus snapshot/query time, source hashes and permissions, parser, chunker/overlap, embedding/prefix, sparse analyzer, ANN parameters, retrievers/fusion/reranker, top-k, context selector/budget/order, model, prompt/schema, decoding/seed, caches, judge, qrels, hardware, and code commit. Store per-query retrieved text/scores, decisions, output claims/citations, latency, cost, and errors. A model family name without revision is not a reproducible configuration.
from rag_evolution.operations import configuration_fingerprint, release_manifest
release = release_manifest(
"corpus-sha-91", "docling-2.4", "section-child-3", "embed-r17",
"hnsw-m32-ef200", "crossencoder-r8", "generator-r12", "prompt-r31"
)
experiment = {
**release,
"fusion": {"method": "rrf", "constant": 60},
"candidate_k": 80,
"rerank_k": 12,
"context_tokens": 6000,
"qrels": "product-gold-2026-08",
"seed": 17,
}
print("Release:", release["release_id"])
print("Experiment:", configuration_fingerprint(experiment))
Release: d6aa3d97e5675f67a2b1d0413b222e2f255bc97edd62b7b3f40ca8c7b963b4ae Experiment: 46af201c8bbf0b27d586c9d0b305bf6a3ba9a830094b3def684d8dd3cc79bfbb
10. Release, rollout, monitoring, and rollback
Offline gates: deterministic unit/golden tests; retrieval/citation/safety thresholds; deletion/ACL/freshness tests; adversarial corpus tests; paired confidence; latency/cost/capacity bounds. Online: shadow, canary by tenant, A/B or interleaving where valid, kill switch, rollback compatible with index schema. Monitor data/parser/chunk/embedding/query/score/route/output drift, support/citation/abstention, incidents, SLOs, and business outcomes.
Maintain runbooks for source outage, stale index, ANN corruption, model/API outage, cache poisoning, cross-tenant leak, deletion failure, cost runaway, prompt injection, and bad rollout. Practice restore and replay.
This notebook is the operational acceptance test, not a leaderboard. A system ships only when its task slices, evidence guarantees, security gates, and resource envelope meet the product contract.
import ast
import json
import re
notebook_files = sorted((ROOT / "notebooks").glob("*.ipynb"))
module_files = sorted((ROOT / "src" / "rag_evolution").glob("*.py"))
test_files = sorted((ROOT / "tests").glob("test_*.py"))
test_count = 0
for path in test_files:
tree = ast.parse(path.read_text(encoding="utf-8"))
test_count += sum(
isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name.startswith("test_")
for node in ast.walk(tree)
)
print("Notebook artifacts:", [path.name for path in notebook_files])
print("Reference modules:", [path.stem for path in module_files])
print("Discovered test methods:", test_count)
print("Handbook Markdown files:", len(list((ROOT / "research").glob("*.md"))))
Notebook artifacts: ['00_complete_rag_handbook.ipynb', '01_rag_evolution.ipynb', '02_advanced_rag.ipynb', '03_evaluation_and_failure_analysis.ipynb', '04_corpus_chunking_and_indexes.ipynb', '05_training_query_fusion_and_reranking.ipynb', '06_structured_multimodal_and_graph_rag.ipynb', '07_agents_memory_temporal_and_security.ipynb', '08_production_evaluation_and_cost.ipynb'] Reference modules: ['__init__', 'agentic', 'chunking', 'context', 'demo_data', 'evaluation', 'generation', 'indexes', 'ingestion', 'memory', 'models', 'operations', 'pipeline', 'rerankers', 'retrievers', 'security', 'selection', 'structured', 'temporal', 'text', 'training'] Discovered test methods: 129 Handbook Markdown files: 18