Skip to article
The Evidence Path
Reader Systems Research Notebooks Python PDF

RAG coverage and verification matrix

18 of 18 · 4,206 words

This matrix is the repository's completeness contract. It maps each material RAG subsystem to the detailed chapter, executable notebook, reference code, and validation method that covers it. “Executable” means a real, dependency-free algorithm or control contract runs locally. “Analytical” means the handbook documents the method, evidence, equations, trade-offs, and evaluation design but does not pretend to reproduce a large neural model, proprietary service, GPU kernel, or billion-scale index.

No finite review can promise that no paper, product, or future technique exists outside it. The stronger and testable claim is that every lifecycle stage and major technique family in the field map has an explicit home, and every intentionally non-reproduced component is labeled.

1. Scope, history, and research method

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Definition and boundaries of RAGField map §1–2, mathematical primer01, complete handbookmodels.py, pipeline contractsAnalytical + executable frame
Source/task/unit/index/policy taxonomiesField map §3–1300Matrix validation in validate_research.pyAnalytical
Historical development before the term RAGChronology, chronological index00, 01203-entry source registryPrimary-source synthesis
2024–2026 frontier and publication-status disciplineFrontier00, 07sources.json status/date validationPrimary-source synthesis
Evidence, claim, chronology, and reproducibility policyResearch method00, 08Link/source/schema validationVerified artifact policy
Terminology and ambiguous namesGlossary00Glossary-term thresholdAnalytical

2. Corpus acquisition, parsing, and governance

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Connectors, crawl/API/file/event ingestionCorpus §204 §1–3CorpusManifest input contractAnalytical contract
Snapshots, CDC, logical IDs, content IDs, versionsCorpus §3, production §104 §1–3CorpusManifest, ManifestEntry, snapshot_hashExecutable
Event time, valid time, system/transaction timeAgents/time §2107 §5–6TemporalFact, BitemporalStoreExecutable
HTML/DOM parsing and boilerplate/active contentCorpus §4.1, security §504 §2, 07 §7strip_active_html, injection inspection testsPartial executable; production parser analytical
PDF reading order, layout, OCR, tables, equationsCorpus §4.2–4.404 §2, 06 §5–7Parser evaluation contractAnalytical; no misleading toy PDF parser
Slides, spreadsheets, code, image, audio, and video parsingCorpus §4.5–4.7, structured §13–1506 §7–8Modality evidence-unit/citation contractAnalytical
Unicode/whitespace normalizationCorpus §504 §2canonicalize_text; exact-offset testsExecutable
Exact and near-duplicate policyCorpus §5.3, security §404 §2, 07 §9ExactDeduplicator, NearDuplicateIndex, near_duplicate_clustersExecutable
Metadata schema and immutable provenanceCorpus §6, security §1604 §1, 07 §8–9content hashes, HMAC provenance, evidence envelopeExecutable teaching contract
ACLs, tenant isolation, trust domainsCorpus §14, security §704 §3, §6, 07 §8ACLPolicy, pre-top-k filters, authorize_resultsExecutable
Licensing, copyright, purpose limitation, retentionSecurity §19–2107 §10Governance/release checklistAnalytical/policy-specific
Correction, deletion, tombstones, unlearning propagationCorpus §15, security §2004 §3, 07 §4manifest and memory tombstone testsExecutable lifecycle core

3. Retrieval units and chunking

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Fixed token windows and overlapCorpus §7.101, 04 §4text.chunk_document offset/overlap testsExecutable
Sentence-aware packingCorpus §7.204 §4sentence_chunks exact character/token lineageExecutable
Paragraph/section/DOM-structure chunksCorpus §7.304 §4section_chunks, Markdown path testsExecutable for Markdown; other parsers analytical
Semantic boundary and discourse chunkingCorpus §7.404 §4Evaluation protocol and substitution seamAnalytical/model-dependent
Proposition/atomic-fact indexingCorpus §7.504 §4Claim/evidence selection interfacesAnalytical/model-dependent extraction
Parent–child/small-to-big retrievalCorpus §7.6, structured §1004 §5, 06 §4parent_child_chunks, hierarchy selection testsExecutable
Contextual prefixes, late chunkingCorpus §7.7, retrieval §5–604 §4, §7Embedding contract and ablation designAnalytical/model-dependent
Recursive/hierarchical summariesCorpus §7.8, structured §1006 §4HierarchyNode, leaf-coverage selectorExecutable selector; learned summaries analytical
Table rows/cells, page regions, code symbols, media segmentsCorpus §7.9, structured §11–1506 §5–8TableRow; modality provenance matrixTable retrieval executable; others analytical
Chunk boundary/recall/duplication/cost evaluationCorpus §804 §4–5, 08 §5lineage, token budgets, evidence flowExecutable

4. Indexes and first-stage retrieval

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Boolean, phrase, proximity, analyzersRetrieval §3.104 §6Inverted-index execution substrateAnalytical beyond unigram lab
TF–IDF/vector-space and query likelihoodChronology, retrieval §3.2, math §1301formula/tests via retrieval metricsAnalytical + adjacent executable
BM25, BM25F, BM25 variantsRetrieval §3.3–3.4, math §201, 04 §6scan BM25 and postings-list InvertedIndexExecutable BM25; fielded variants analytical
Pseudo-relevance feedback/Rocchio/relevance modelsRetrieval §3.5, math §1305 §5Transformation experiment contractAnalytical
WAND/Block-Max WAND and top-k pruningRetrieval §3.604 §6Postings exposed; exact semantics documentedAnalytical optimization
DeepCT/doc2query/DeepImpact/uniCOIL/COILRetrieval §4.1–4.305 §4Source/evaluation coverageAnalytical neural models
SPLADE and learned sparse regularizationRetrieval §4.4, training §704 §6, 05 §4Loss/index-growth discussion and metricsAnalytical neural model
Dense dual encoders: DPR, ANCE, Contriever, GTR, E5, DRAGONRetrieval §5, chronology01, 04 §7, 05 §4exact cosine index; objective calculationsExecutable interface/math; neural checkpoints analytical
Late interaction: ColBERT, PLAID, XTR, CITADELRetrieval §6, math §406 §6late_interaction_score, pooling testsExecutable MaxSim core
Exact dense search oracleCorpus §1204 §7–8ExactCosineIndexExecutable
IVF coarse quantizationCorpus §13.2, math §1704 §8IVFCoarseIndex, exact recall auditExecutable
PQ/OPQ/scalar quantizationCorpus §13.3, math §1704 §9ScalarQuantizer, ProductQuantizer, distortion/storage auditExecutable PQ/scalar; OPQ analytical
HNSWCorpus §13.4, math §1804 §8Algorithm/evaluation contractAnalytical; no toy scale claim
DiskANN/Vamana, SPANN, ScaNNCorpus §13.5–13.704 §8Resource/recall evaluation designAnalytical systems algorithms
ANN recall, filters, latency, storage, updatesCorpus §13–14, retrieval §704 §8–10evaluate_ann_recall, pre-top-k filters, quantization auditExecutable

5. Query understanding, fusion, and ranking

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Intent classification and entity/temporal/metadata extractionRetrieval §205 §5, 07 §2AdaptiveRetriever routingExecutable proxy + analytical production design
Spelling/normalization/aliasesRetrieval §8.101, 05 §5canonical/synonym transformsExecutable basic transform
Multi-query/RAG-FusionRetrieval §8.302, 05 §5–6MultiQueryRetriever, RRFExecutable
HyDE, Query2Doc, generated expansionRetrieval §8.3–8.405 §5Ablation/logging contractAnalytical model-dependent generation
Conversation rewriting, decomposition, step-backRetrieval §8.5–8.707 §1–2iterative planner/traceExecutable simple decomposition; learned rewriting analytical
RRFRetrieval §9.1, math §701, 05 §6reciprocal_rank_fusion, HybridRetrieverExecutable
Score calibration, CombSUM/CombMNZRetrieval §9.2–9.4, math §1605 §6min-max/z-score calibrated CombSUMExecutable CombSUM; learned/calibrated variants analytical
Cross-encoder, monoT5/RankT5, RankGPT/listwise rerankingRetrieval §10, training §902, 05 §7CrossFeatureReranker; point/pair/listwise objectivesExecutable boundary/math; neural rerankers analytical
MMR/deduplication/diversityRetrieval §11.1, context §2–302, 05 §8ContextPackerExecutable
Weighted set cover/knapsack/claim coverageRetrieval §11.2, math §1905 §8greedy_budgeted_coverage, selection regretExecutable
Authority, independence, time, contradiction-aware selectionRetrieval §11.3–11.505 §8, 07 §5–9trust domains, bitemporal lookup, duplicate clustersExecutable primitives + analytical policies
Sufficiency/calibration/stoppingRetrieval §12, context §1702, 07 §2abstention thresholds, bounded stop conditionsExecutable proxy + analytical calibration

6. Context and generation

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Context budgeting, deduplication, ordering, serializationContext §1–502, 05 §8–10ContextPacker, untrusted evidence envelopeExecutable
Extractive, abstractive, token, and latent compressionContext §606 §4, §6hierarchy/vector pooling; evaluation contractPartial executable; learned compressors analytical
Prompt-based context-only/permissive RAGContext §702extractive cited generatorExecutable policy proxy
Original RAG-Sequence/RAG-Token latent integrationContext §8, chronology, math §501formal objective and historical result auditAnalytical neural architecture
FiD/multi-passage readersContext §9, chronology01mechanism/evaluation coverageAnalytical neural architecture
REALM, RETRO, Atlas, kNN-LM/retrieval pretrainingContext §10, chronology00, 01objectives, dates, results, limitsAnalytical neural systems
Frozen/black-box generator RAGContext §1102modular pipeline seamExecutable interface
Iterative generation: IRCoT, FLARE, ITER-RETGEN, Self-RAGContext §12, agents §5–1107 §1–2budgeted iterative trajectoryExecutable control proxy + analytical methods
Long-form claim decomposition and citationsContext §14–15, evaluation §602, 08 §2–5source-linked Citation, claim/citation metricsExecutable basic attribution + analytical NLI/human audit
Verification/correction loopsContext §1602, 08 §5trace/failure attributionPartial executable
Abstention, uncertainty, conflicts, numerical and temporal claimsContext §17–1802, 07 §5–6pipeline abstention and temporal ambiguityExecutable core
Structured/constrained outputsContext §19, production §1008 §1schema/version/retry contractAnalytical
Long context versus retrieval and Self-RouteContext §21, field map §1407 §2route evaluation contractAnalytical

7. Learning and optimization

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Positive-label provenance and biasTraining §1–305 §1label contract/checklistAnalytical
Random/in-batch/cross-batch/BM25/ANN/teacher/adversarial negativesTraining §405 §2–3NegativeExample, mine_hard_negativesExecutable policy core
False-negative maskingTraining §4.805 §2false_negative_mask, masked loss testsExecutable
InfoNCE/multiple-negative softmaxTraining §5.1, math §1405 §1–2contrastive_loss, in_batch_contrastive_lossExecutable math
Triplet/hinge/pairwise logistic/listwise lossesTraining §5.2–5.405 §1, §4hinge, logistic, listwise implementationsExecutable math
Margin-MSE and KL/listwise distillationTraining §5.5–5.6, math §1505 §4kl_distillation_lossExecutable KL core; margin-MSE analytical
Retrieval-oriented pretraining and learned sparse/multi-vector trainingTraining §6–805 §4primary-source/objective auditAnalytical neural training
Reranker/rewriter/decomposer trainingTraining §9–1005 §5–7ranking objectives and trace evaluationExecutable objectives; models analytical
Generator SFT, evidence dropout, citation trainingTraining §1105 §9–10curriculum/release protocolAnalytical neural training
Latent-document joint learning, EM, reader-to-retriever distillationTraining §12–14, math §500objective derivationsAnalytical neural training
DPO/preference optimizationTraining §1505 §9dpo_lossExecutable math
RL search: policy gradients, value learning, process/outcome rewardsTraining §16–17, agents §11–1205 §9, 07 §1–2returns/advantages/REINFORCE, bounded agentExecutable math/control; large RL runs analytical
Reward hacking, curriculum, domain/continual learningTraining §17–2105 §9–10adversarial/release checklistAnalytical

8. Structured and multimodal RAG

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Curated KG entity linking/path retrieval/GNN QAStructured §306 §1–2weighted PPR and graph expansionExecutable propagation; neural/entity linking analytical
Passage/entity associative graphs and HippoRAGStructured §402, 06 §1–2metadata graph expansion, PPRExecutable core
Microsoft GraphRAG community reports/DRIFTStructured §506 §3community-selection proxy and cost/eval contractAnalytical LLM extraction/summarization
Query-specific/dynamic graph RAGStructured §606 §1–2query-seeded graph propagationExecutable core
PageRank, path, beam, community, GNN, LLM traversalStructured §706 §1–3exact weighted PPR; other algorithms documentedPartial executable
Graph extraction/entity resolution/confidence/contradictionStructured §8–906 §1–3construction/evaluation contractAnalytical/model-dependent
RAPTOR/structural hierarchy/multi-resolution/map-reduceStructured §1006 §4select_hierarchy with leaf lineageExecutable selector; learned summaries analytical
Table serialization, row/cell retrieval, SQL/program executionStructured §1106 §5retrieve_table_rows with numeric/schema featuresExecutable row retrieval; governed SQL analytical
Visual documents: text-first, captions, ColPali, VisRAG, VDocRAGStructured §1206 §6patch MaxSim and pooling/storage trade-offExecutable core; VLMs analytical
Image/chart/map/audio/video RAGStructured §1306 §7modality evidence/citation contractAnalytical
Code/repository RAGStructured §1406 §8symbol/graph/commit provenance designAnalytical
Web/live search and API/tool RAGStructured §15, agents §1306 §8, 07source-time/tool permission contractsAnalytical
Multilingual/cross-lingual RAGStructured §1606 §8language-pair evaluation matrixAnalytical
Biomedical, legal, finance, science, enterprise/personalStructured §17–2106 §8domain-specific authority/time/privacy gatesAnalytical

9. Adaptive search, agents, memory, and time

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
No/one/multi retrieval routing and Adaptive-RAGAgents §3–407 §2AdaptiveRetrieverExecutable deterministic proxy
Self-RAG reflection/control tokensAgents §5, frontier07 §1–2control taxonomy and trace evaluationAnalytical neural model
Corrective/evidence-grading RAGAgents §607 §1–2retry/stop policy seamPartial executable
FLARE/uncertainty-triggered active retrievalAgents §707 §1–2bounded policy contractAnalytical neural uncertainty
ReAct, Self-Ask, IRCoT and multi-hop stateAgents §8–1007 §1BudgetedIterativeRetriever, plan/accumulate/stop traceExecutable control proxy
Search-R1, ReSearch, StepSearch, GRIP, Q-RAG, DeepRAG, HiPRAGAgents §11–12, frontier07 §1–2reward/control comparisonAnalytical peer-reviewed frontier
Source/retriever/tool/long-context routingAgents §13–1407 §2router and regret/route metricsExecutable proxy + analytical policies
Working/episodic/semantic/procedural/profile/latent memoryAgents §15–1607 §3typed MemoryRecord kindsExecutable external types; latent memory analytical
Memory write/retrieve/update/supersede/consolidate/forgetAgents §17–1907 §3–4MemoryStore, write policy, groups, tombstonesExecutable
LongMemEval and memory metricsAgents §2007 §3–4lifecycle metric contractAnalytical benchmark application
Temporal scoring/version conflict/freshnessAgents §21–22, math §2007 §5–6bitemporal lookup, decay, stale rateExecutable
Temporal/permission/model-sensitive cachesAgents §23, production §907 §6, 08cache_identityExecutable key contract

10. Grounding, evaluation, and statistics

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Retrieval precision/recall/MRR/nDCGEvaluation §2, math §1101, 03, 08 §2metric implementations/testsExecutable
Answer EM/token F1/claim correctness/completenessEvaluation §303, 08 §3EM/F1; claim evaluation designPartial executable
Faithfulness/context utilizationEvaluation §403, 08 §2–3lexical faithfulness explicitly labeled proxyExecutable proxy + analytical human/NLI audit
Citation precision/recall/entailment/completeness/authorityEvaluation §5–6, context §14–1502, 03, 08source-linked citations, precision/recallPartial executable; entailment/authority human audit analytical
Abstention/selective prediction/calibrationEvaluation §7, context §1702, 03abstention behavior/testsExecutable proxy
RAGAS, ARES, RAGChecker, RAGTruth/RAGBenchEvaluation benchmark audit08 §2taxonomy, assumptions, use guidanceAnalytical evaluator audit
KILT, BEIR, MTEB/MMTEB, BRIGHTEvaluation benchmark audit08 §2benchmark/metric compatibility mapAnalytical benchmark audit
RGB, CRUD-RAG, CRAG, NoMIRACL, mtRAG, TREC RAGEvaluation benchmark audit, frontier08 §2targeted risk-suite guidanceAnalytical benchmark audit
Oracle/closed-book/distractor/end-to-end controlsEvaluation protocol08 §3layered experiment designExecutable end-to-end; oracle setup analytical
Per-query slices and failure attributionEvaluation, retrieval §1503, 08 §4–5metrics_by_tag, evidence_flowExecutable
Paired bootstrap intervals and human/judge auditEvaluation statistics03, 08 §4paired_bootstrap_deltaExecutable bootstrap; human/PPI analytical

11. Security, privacy, and governance

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Assets, actors, trust boundaries, threat modelSecurity §1–307 §7–10explicit untrusted evidence envelopeAnalytical + executable boundary
Corpus/memory poisoning and trigger backdoorsSecurity §407 §9near-duplicate clusters and canariesExecutable diagnostics; attacks/guarantees analytical
Indirect prompt injection and tool coercionSecurity §507 §7active-content removal, multi-signal inspectionExecutable detector; explicitly not a guarantee
Retrieval manipulation/source spoofingSecurity §607 §8–9trust domains, signed provenanceExecutable teaching controls
Cross-tenant leakage and ACL bypassSecurity §707 §8tenant/ACL denial testsExecutable
Corpus extraction/membership inference/embedding-query privacySecurity §8–1007 §10threat/evaluation matrixAnalytical privacy research
PIR, encrypted/TEE/federated privacy-preserving retrievalSecurity §1100design/performance trade-off auditAnalytical cryptographic/systems methods
Multimodal/graph privacy and cache side channelsSecurity §12–1307 §10threat and cache-identity contractsAnalytical + executable cache key
Safety degradation and RAG-specific red teamingSecurity §14, §2507 §7–10attack matrix/checklistAnalytical
Resource/DoS attacksSecurity §15, production §7–807 §7, 08 §7oversized-content signal and hard budgetsExecutable basic controls
Citation/provenance integritySecurity §1607 §8–9hashes, HMAC, immutable IDsExecutable teaching controls
Filtering, conflict-aware, certified/conformal defensesSecurity §17–18, frontier07 §9assumptions/limits and aggregation diagnosticsAnalytical guarantees; executable diagnostics
Supply chain, logging privacy, secure releaseSecurity §22–2407 §10, 08 §9–10release fingerprint and governance gatesExecutable release identity + analytical controls
Incident response and residual human controlSecurity §26–27, production §2507 §10, 08 §10runbook checklistAnalytical/organization-specific

12. Production systems and economics

SubjectDetailed materialNotebookExecutable or validation evidenceCoverage level
Data/ingest/control/evaluation/serving planesProduction §108 §1versioned component contractsAnalytical architecture
Auth → route → retrieve → rerank → pack → generate → verify request pathProduction §202, 08 §1pipeline TraceEventsExecutable
Quality/latency/availability/freshness/security/cost SLOsProduction §308 §6–8p50/p95/p99, cost/failure/cache summariesExecutable
Capacity model and queuing limitsProduction §408 §6–7calls/tokens/latency budget modelPartial executable
Sparse/dense/multi-vector/federated servingProduction §504, 06local reference indexes; production topology analyticalPartial executable
Sharding, replicas, batching, scheduling, backpressureProduction §6–708 §6–7SLI/capacity designAnalytical distributed systems
Context/tool/token/time/cost hard budgetsProduction §807 §1–2, 08 §7ServiceBudget, check_budget, bounded agentExecutable
Source/parse/embedding/retrieval/answer/KV cachesProduction §907 §6, 08 §6cache identity/hit metricsExecutable key/metrics; cache engines analytical
Generation serving, prefill/decode, speculative retrievalProduction §1008 §6stage metrics/evaluation contractAnalytical serving kernels
Dependencies, retries, circuit breakers, degraded modesProduction §11–1208 §7, §10budget/rollback/runbook contractAnalytical
Trace schema and online monitoringProduction §13–1508 §1, §6RequestMeasurement, StageMeasurement, stage summariesExecutable
Query/corpus/model/index/evaluator driftProduction §1608 §4, §9–10slice/release comparison protocolAnalytical + executable primitives
Offline/shadow/canary/rollback and feature flagsProduction §17–1808 §9–10content-addressed release manifestExecutable identity; deployment analytical
End-to-end cost accountingProduction §19, math §2108 §6–8per-stage/request costExecutable
Efficiency: datastore, pruning, ANN, pooling, batching, early stopProduction §20, frontier04 §8–9, 06 §6, 08 §6–8recall/storage/latency/cost trade-offsExecutable representative levers
Vector/search platform and build-vs-buy selectionProduction §21–2208 §8–10requirement/Pareto frameworkAnalytical/vendor-neutral
Multi-region/DR/testing/runbooks/readinessProduction §23–2908 §9–10release and incident checklistsAnalytical/organization-specific
Constrained utility and Pareto frontierProduction §19–20, math §2108 §8SystemCandidate, pareto_frontier, constrained_choiceExecutable

13. Verification inventory

The current automated gate checks:

  • all handbook and notebook artifacts exist and internal links resolve;
  • the source registry has unique IDs/URLs, valid statuses, dates, topics, and a minimum breadth threshold;
  • the handbook exceeds a minimum narrative/primary-link threshold and contains no unfinished-marker placeholders;
  • the complete field-map topics appear in this matrix;
  • every required notebook is valid nbformat 4, uses a Python kernel, has been executed, contains no error output, and retains saved output;
  • corpus documents preserve dates, resolvable sources, and graph metadata;
  • unit/integration tests cover retrieval, chunk lineage, indexes, quantization, training objectives, fusion/selection, graph/table/visual primitives, memory, bitemporal lookup, security controls, SLOs, budgets, citations, and abstention;
  • notebook regeneration and execution are deterministic;
  • source distributions, research word count, notebook cell counts, glossary breadth, and coverage-row count are printed for inspection.

The intentionally non-reproduced surfaces—large neural training, proprietary models, distributed search engines, cryptographic private retrieval, and billion-scale benchmarks—are documented with their objectives, assumptions, primary evidence, costs, and a replacement/evaluation contract. A small local simulation is never presented as evidence for their published performance.

← RAG glossary
Typesetting mathematics…
The Evidence Path · evidence cutoff 9 August 2026