Research question
How did retrieval-augmented generation evolve, what technical problems drove each transition, and which technique families define the state of the art as of 2026-08-09?
The phrase “state of the art” is treated as a conditional claim. A system may be frontier-quality for multi-hop evidence discovery, visual-page retrieval, global corpus synthesis, citation completeness, or accuracy/cost trade-offs without dominating the other dimensions.
Evidence policy
The review uses this source priority:
- final peer-reviewed paper in official proceedings;
- accepted-paper page or author manuscript when proceedings are unavailable;
- original preprint for work not yet formally published;
- first-party technical report for influential industry systems;
- official repository or dataset card for implementation details not stated in the paper.
Surveys help discover terminology and references but do not support mechanism, result, or priority claims. Secondary blog posts and vendor comparisons are not used as independent evidence. The machine-readable sources.json labels each source as peer-reviewed, preprint, industry-report, or benchmark-program.
The mathematical primer collects the objectives and systems assumptions that recur across papers so the chronology can focus on empirical and historical differences.
The field map is the scope contract, the chronological index lists every registered work by first-public date, and the coverage matrix maps every major subject to its detailed chapter, Jupyter lab, reference implementation, and verification evidence. The six continuous-prose field_notebook folios provide the book-length reading path; the complete Jupyter edition places those expressive folios in front of the entire technical atlas in notebooks/00_complete_rag_handbook.ipynb.
Chronology policy
Both first public date and formal publication matter. For example, FiD and FiD-KD were public in 2020 but published in 2021; RETRO was public in 2021 and published in 2022; Atlas was a 2022 preprint and a 2023 JMLR article. The chronology orders first public disclosure and reports the venue separately.
This prevents two common historical errors:
- pretending the term “RAG” created retrieve-then-generate architectures; and
- moving methods by a year or more merely because one cites the conference date and another cites the arXiv date.
Claim policy
Every method entry separates:
- mechanism: representation, retrieval, integration, and policy;
- learning signal: labels, contrastive loss, marginal likelihood, distillation, supervised trajectories, or reinforcement learning;
- evidence: corpus, dataset, metric, and the original paper's result;
- limitations: threats to validity and deployment constraints;
- influence: the architectural idea inherited by later work.
Reported scores are not recomputed in this repository. They are attributed to their source and are not compared across incompatible settings. Phrases such as “state of the art” are retained only when scoped to the paper's own benchmark and setting; the synthesis prefers exact metrics and caveats.
Reproduction policy
For an experiment to be replayable, record at least:
- corpus source, timestamp, content hash, permissions, and deletion state;
- document parser, chunker, overlap, metadata, embedding and index versions;
- retriever, fusion, reranker, top-\(k\), score calibration, and ANN parameters;
- generator, prompt, context order, context budget, decoding, and random seed;
- every retrieved chunk with score, rank, source ID, and retrieval timestamp;
- answer claims, citation spans, abstention/confidence, latency, token count, and cost;
- evaluation set/qrels version, judge prompt/model, human adjudication, and uncertainty interval.
Mutable web search and closed model APIs require an immutable trace. A model name such as “GPT-4” or “Claude” without an exact revision/date is insufficient for a durable result.
What the executable code proves—and does not
The code verifies algorithmic ideas at small scale: versioned manifests, content hashes, exact/near deduplication, ACLs and tombstones; fixed, sentence, section, and parent–child chunks with exact lineage; postings-list BM25; exact cosine and IVF search; scalar/product quantization and ANN recall; RRF and calibrated score fusion; multi-query and graph expansion; adaptive/iterative routing; interaction reranking; MMR and claim-coverage selection; InfoNCE, pairwise/listwise/distillation/DPO/REINFORCE objectives; Personalized PageRank, table retrieval, visual late interaction, and hierarchical selection; memory write/retrieve/update/consolidate/delete; bitemporal lookup and freshness; authorization, active-content/injection signals, provenance signatures, poison-cluster/canary diagnostics; citations, abstention, layered metrics, paired bootstrap intervals, SLOs, budgets, cost, and Pareto selection.
It does not reproduce billion-parameter training runs or paper leaderboards. The hashed semantic retriever, heuristic reranker, learned-model boundaries, and security detectors are deliberately labeled teaching proxies. Distributed ANN engines, neural parsers/encoders/readers, VLMs, private-retrieval protocols, and certified defenses are documented with substitution and evaluation contracts rather than misrepresented by trivial demos. Every local decision is inspectable and every notebook runs without model downloads, credentials, or mutable external services.