# A chronological technical history of retrieval-augmented generation **Coverage:** conceptual foundations through 2023, with 2024–2026 continued in the [frontier review](frontier_2024_2026.md). Dates below use first public release; formal venues are shown separately. ## Chronology at a glance | First public | Formal venue | Milestone | Architectural shift | |---|---|---|---| | 1972–1976 | journals | TF-IDF, vector-space retrieval, relevance weighting | corpus statistics and sparse lexical matching | | 1994/1995 | TREC-3 | Okapi BM25 | probabilistic term weighting with saturation and length normalization | | 2014-10-15 | ICLR 2015 | Memory Networks | raw facts as addressable neural memory | | 2015-03-31 | NeurIPS 2015 | End-To-End Memory Networks | differentiable soft multi-hop reads from answer supervision | | 2017-03-31 | ACL 2017 | DrQA | Wikipedia-scale retrieve-then-read | | 2018 | ICLR/EMNLP workshops | Wizard of Wikipedia; Retrieve and Refine | retrieval-conditioned dialogue generation | | 2019-06-01 | ACL 2019 | ORQA | latent dense retrieval with answer-only supervision | | 2019-11-01 | ICLR 2020 | kNN-LM | token-level non-parametric language-model memory | | 2020-02-10 | ICML 2020 | REALM | retrieval trained during language-model pretraining | | 2020-04-10 | EMNLP 2020 | DPR | simple supervised bi-encoder retrieval at Wikipedia scale | | 2020-05-22 | NeurIPS 2020 | RAG | latent-document retrieval plus pretrained seq2seq generation | | 2020-06-26 | NeurIPS 2020 | MARGE | retrieve-related-documents pretraining from scratch | | 2020-07-02 | EACL 2021 | FiD | independent passage encoders, joint decoder fusion | | 2020-12-08 | ICLR 2021 | FiD-KD | distill reader attention into the retriever | | 2021 | TACL 2021 | SPALM | learn a context-dependent parametric/memory gate | | 2021-02/03 | NAACL 2021 | KILT | shared snapshot and provenance-gated evaluation | | 2021-06-09 | NeurIPS 2021 | EMDR² | joint latent multi-document retriever-reader training | | 2021-09-20 | — | SPLADEv2 | learned sparse expansion in an inverted index | | 2021-12-08 | ICML 2022 | RETRO | chunk retrieval from a trillion-token datastore during LM training | | 2021-12-16 | TMLR 2022 | Contriever | unsupervised contrastive dense retrieval | | 2021-12-03 | NAACL 2022 | ColBERTv2 | compressed token-level late interaction | | 2022-08-05 | JMLR 2023 | Atlas | few-shot retrieval-augmented pretraining and reader-to-retriever learning | | 2022-10-06 | ICLR 2023 | ReAct | reasoning interleaved with search/tool actions | | 2022-10-06 | EMNLP 2022 | MuRAG | end-to-end text-and-image retrieval augmentation | | 2022-11-22 | — preprint | RA-CM3 | retrieved multimodal documents for generative modeling | | 2022-12-20 | ACL 2023 | HyDE | generated hypothetical documents as zero-shot queries | | 2023-03-14 | EMNLP 2023 | Query2Doc | LLM pseudo-documents expand sparse and dense queries | | 2023-05-11 | EMNLP 2023 | FLARE | retrieve while generating when future tokens are uncertain | | 2023-05-23 | EMNLP 2023 | Rewrite–Retrieve–Read | train a query rewriter from downstream answer reward | | 2023-05-24 | — preprint | ITER-RETGEN | alternate complete generations and retrieval | | 2023-07-06 | TACL 2024 | Lost in the Middle | demonstrate positional failure in long contexts | | 2023-10-17 | ICLR 2024 | Self-RAG | retrieval and evidence critique as generated reflection tokens | The table is not one lineage. Four branches converge around 2020: 1. **sparse retrieve/read:** BM25 → DrQA; 2. **differentiable memory and latent evidence:** Memory Networks → MemN2N → ORQA → REALM; 3. **dense retrieval plus sequence generation:** DPR → RAG / FiD → FiD-KD / EMDR² → Atlas; 4. **non-parametric language modeling:** kNN-LM → SPALM → RETRO. The 2022–2023 systems add a fifth branch: **inference-time retrieval control**, from query generation and rewriting to uncertainty triggers, iteration, tools, and model-generated evidence judgments. --- ## 1. Before “RAG”: probabilistic information retrieval ### 1972–1976: term specificity, vector space, and relevance odds Karen Spärck Jones's 1972 term-specificity paper formalized the intuition behind inverse document frequency: a term occurring in few documents is more discriminative than one occurring everywhere. Salton, Wong, and Yang's 1975 vector-space model represented queries and documents as weighted term vectors and ranked by vector similarity. Robertson and Spärck Jones's 1976 relevance weighting derived term weights from probabilistic relevance odds. With \(N\) documents, \(n_t\) containing term \(t\), \(R\) judged relevant, and \(r_t\) relevant documents containing \(t\), the Robertson–Spärck Jones weight is \[ \log \frac{(r_t+0.5)/(R-r_t+0.5)} {(n_t-r_t+0.5)/(N-n_t-R+r_t+0.5)}. \] Without relevance judgments, it reduces to an IDF-like prior. The essential engineering invention was the inverted index: map a term to the documents and positions containing it, rather than scan every document. ### 1994–1995: Okapi BM25 Robertson, Walker, Jones, Hancock-Beaulieu, and Gatford described Okapi's TREC-3 experiments; the mature BM25 account was later consolidated by Robertson and Zaragoza. A common score is \[ \operatorname{BM25}(q,d)= \sum_{t\in q} \operatorname{IDF}(t) \frac{(k_1+1)f(t,d)} {f(t,d)+k_1\left(1-b+b|d|/\operatorname{avgdl}\right)}. \] - \(k_1\) controls term-frequency saturation: the tenth occurrence contributes less than the first. - \(b\) controls document-length normalization. - implementations may add query-term saturation, field weights, proximity, or a positive-IDF variant. BM25's limitations—exact-term dependence, vocabulary mismatch, and no task objective—motivated dense retrieval. Its advantages never disappeared: rare identifiers survive, indexing and incremental updates are cheap, scores can be explained, and postings filters compose naturally with access control and metadata. DrQA used hashed TF-IDF; DPR used a BM25 hard negative; DPR itself lost to BM25 on SQuAD; modern high-recall stacks often fuse sparse and dense results. **Historical caveat.** The original TREC-3 paper combined weighting, passage retrieval, expansion, and routing changes. It does not isolate a portable “BM25 gain,” so later leaderboard scores should not be retrospectively attributed to that experiment. Primary sources: [Spärck Jones 1972](https://doi.org/10.1108/eb026526), [vector-space model 1975](https://doi.org/10.1145/361219.361220), [relevance weighting 1976](https://doi.org/10.1002/asi.4630270302), [Okapi at TREC-3](https://pages.nist.gov/trec-browser/trec3/proceedings/), and [BM25 and Beyond](https://doi.org/10.1561/1500000019). --- ## 2. Explicit neural memory and retrieve-then-read ### 2014: Memory Networks Jason Weston, Sumit Chopra, and Antoine Bordes defined a system with input, memory-update, output, and response modules \((I,G,O,R)\). Statements occupy external memory slots. One or two supporting memories are selected with hard maximum-score reads: \[ o_1=\arg\max_i s_O(x,m_i),\qquad o_2=\arg\max_i s_O([x,m_{o_1}],m_i). \] Bilinear embedding scores learn the match. Separate margin-ranking objectives train each evidence hop and the final one-word response. The paper proposes an RNN response generator, but its main evaluated system ranks answer words; it is not a modern free-form RAG model. The large experiment stored roughly 14 million ReVerb facts from ClueWeb09 plus QA and WikiAnswers data. Embedding-only F1 was 0.72; exact bag-of-words features raised it to 0.82. Cluster hashing reduced an average search from 14 million to 177,000 candidates at 0.80 F1; word hashing reduced it to 13,000 at 0.68. A time-aware two-hop model nearly solved synthetic reasoning tasks that RNN and LSTM baselines did not. The system required supporting-fact labels; hard retrieval was non-differentiable; the best multi-hop evidence was synthetic. Its durable idea was raw, addressable, repeatedly readable external memory—a direct conceptual ancestor of RAG. [Original paper](https://arxiv.org/abs/1410.3916). ### 2015: End-To-End Memory Networks Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus replaced hard reads with soft attention. For memory item \(x_i\) and query \(q\), \[ m_i=Ax_i,\quad c_i=Cx_i,\quad u=Bq, \] \[ p_i=\operatorname{softmax}(u^\top m_i),\quad o=\sum_i p_i c_i,\quad u^{k+1}=u^k+o^k. \] Answer cross-entropy backpropagates through every read, eliminating explicit supporting-fact labels. Position and temporal encodings preserve word order and recency; weight tying controls parameters; multiple hops permit composition. On bAbI, the narrative reports best mean error of 12.6% with 1,000 examples and 4.2% with 10,000, compared with the strongly supervised Memory Network's 6.7% and 3.2%. The main-table 12.4%/7.5% values describe particular joint variants, not the overall best. Language-model experiments were competitive with then current LSTMs, but memory covered only about 50 sentences or 100–200 words and softmax still scanned every slot. Ten random restarts and strong initialization sensitivity further limit the result. The influence is the differentiable, weakly supervised multi-hop read later paired with large ANN indexes. [Official paper](https://proceedings.neurips.cc/paper/2015/hash/8fb21ee7a2207526da55a679f0332de2-Abstract.html). ### 2017: DrQA operationalizes Wikipedia as memory Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes built a two-stage open QA system over the 2016-12-21 English Wikipedia dump: - 5,075,182 articles; - hashed unigram/bigram TF-IDF using MurmurHash into \(2^{24}\) bins; - an inverted index returning five articles; - a three-layer bidirectional LSTM reader predicting answer-span start and end. Retriever and reader were trained independently. For datasets without evidence, distant supervision kept retrieved paragraphs containing an answer string. Top-five answer-string recall was 77.8% on SQuAD, 86.0% on CuratedTREC, 74.4% on WebQuestions, and 70.3% on WikiMovies. Full-Wikipedia top-one exact match was 29.8, 25.4, 20.7, and 36.5, respectively. On SQuAD, reader-only development EM was 69.5 but the full system reached 27.1, cleanly exposing the retrieval ceiling. DrQA was lexical, extractive, paragraph-local, and pipelined. It nevertheless made large-scale retrieve-then-read standard and explicitly identified multi-passage aggregation and joint retriever-reader learning as future work. [ACL paper](https://aclanthology.org/P17-1171/). ### 2018: retrieval enters neural generation Retrieve and Refine retrieved a training utterance with a Key-Value Memory Network, then conditioned an attentive LSTM on the dialogue and retrieved reply. On ConvAI2, the strongest variant's human engagingness was 3.80/5 versus 3.66 for retrieval and 2.70 for seq2seq. Yet standard retrieval barely moved perplexity, an early warning that likelihood may not reflect retrieval-conditioned generation quality. It retrieved response candidates rather than factual evidence and did not provide provenance. [Paper](https://aclanthology.org/W18-5713/). Wizard of Wikipedia paired knowledge retrieval with knowledge-grounded conversation and human evaluation. These dialogue branches show why “RAG invented retrieve-then-generate in 2020” is historically wrong. What the 2020 RAG paper supplied was a general pretrained seq2seq latent-document formulation and a name that became standard. [Wizard paper](https://openreview.net/forum?id=r1l73iRqKm). --- ## 3. Latent dense retrieval and non-parametric language models ### 2019: ORQA learns dense retrieval from answer strings Kenton Lee, Ming-Wei Chang, and Kristina Toutanova used separate BERT-base query and block encoders projected to 128 dimensions: \[ s_{\text{retr}}(q,b)=h_q^\top h_b. \] A BERT cross-encoder scored answer spans. Training marginalized probability over retrieved spans exactly matching any answer string. The Inverse Cloze Task provided a cold start: treat a sentence as a pseudo-query and its surrounding block as positive evidence. The corpus was the 2018-12-20 English Wikipedia split into just over 13 million blocks of at most 288 wordpieces. A locality-sensitive-hashing MIPS index returned five blocks. Document embeddings stayed fixed during QA fine-tuning. Test EM, BM25+BERT → ORQA: - Natural Questions: 26.5 → 33.3; - WebQuestions: 17.7 → 36.4; - CuratedTREC: 21.3 → 30.1; - TriviaQA: 47.1 → 45.0; - SQuAD: 33.2 → 20.2. Dense semantic retrieval helped genuine information-seeking questions but hurt datasets created while annotators knew the evidence. The 128-dimensional single-vector bottleneck, fixed document side, top-five truncation, and spurious answer-string matches constrained it. ORQA nevertheless established the dense MIPS + latent evidence blueprint. [ACL paper](https://aclanthology.org/P19-1612/). ### 2019: kNN-LM retrieves token contexts at inference Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis stored each training-token context representation as key and the following token as value: \[ (K,V)=\{(f(c_i),w_i)\}. \] At inference, nearest contexts define a token distribution: \[ p_{\text{kNN}}(y\mid x)\propto \sum_{(k_i,v_i)\in N_k} \mathbf 1[y=v_i]\exp[-d(k_i,f(x))], \] \[ p(y\mid x)=\lambda p_{\text{kNN}}(y\mid x)+(1-\lambda)p_{\text{LM}}(y\mid x). \] No additional model training was required. FAISS searched quantized 1,024-dimensional keys, usually with \(k=1024\). On WikiText-103, base test perplexity 18.65 became 16.12 with kNN-LM; 15.79 required adding a separate continuous cache. Using a 3B-token datastore with a model trained on only 100M tokens produced perplexity 13.73, better than a model trained on all 3B at 15.17. Swapping a Books datastore into a Wikipedia-trained model reduced Books perplexity from 34.84 to 20.47. An entry and lookup per token imposed large storage and latency; retrieval and the interpolation weight were not jointly learned; evidence was not document-level or provenance-bearing. Its influence is hot-swappable non-parametric memory for rare facts, leading to adaptive gating in SPALM and chunk retrieval in RETRO. [ICLR paper](https://openreview.net/forum?id=HklBjCEKvH). ### 2020: REALM makes retrieval part of pretraining Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang treated the document \(z\) as latent during masked-language-model pretraining: \[ p(y\mid x)=\sum_{z\in\mathcal Z}p(y\mid x,z)p(z\mid x),\qquad p(z\mid x)\propto \exp(E_x(x)^\top E_z(z)). \] A BERT bi-encoder retrieved; another BERT cross-encoded input and evidence. Documents received positive learning signal when they increased target likelihood. Salient entity/date masking, a null document, exclusion of the source document, ICT initialization, and periodic index refresh prevented easy shortcuts. REALM used the same 13M-block 2018 Wikipedia corpus. It marginalized eight candidates in pretraining and five for QA. Cached embeddings and MIPS were rebuilt asynchronously about every 500 steps; downstream QA froze the document encoder/index while the query side changed. Test EM on Natural Questions/WebQuestions/CuratedTREC was 39.2/40.2/46.8 for Wikipedia pretraining and 40.4/40.7/42.9 for CC-News pretraining, versus ORQA's 33.3/36.4/30.1. A 30-times staler index collapsed NQ development EM from 38.2 to 28.7. Random token masking produced 32.3 versus 38.2 for salient spans. The engineering burden—64 TPUs in reported pretraining, index rebuilding, top-\(k\) approximation, frozen documents at QA time—and extractive output were substantial. REALM is the closest direct conceptual ancestor of RAG: retrieval participates in pretraining, fine-tuning, and inference, and learns from output likelihood. [ICML paper](https://proceedings.mlr.press/v119/guu20a.html). ### 2020: DPR makes dense retrieval simple and modular Vladimir Karpukhin and colleagues trained independent BERT-base encoders by contrastive negative log likelihood: \[ s(q,p)=E_Q(q)^\top E_P(p), \] \[ \mathcal L=-\log \frac{e^{s(q,p^+)}} {e^{s(q,p^+)}+\sum_j e^{s(q,p_j^-)}}. \] The best recipe combined other questions' positives as in-batch negatives with one high-ranked BM25 passage lacking the answer. The 2018 Wikipedia snapshot was split into exactly 21,015,324 non-overlapping 100-word passages; 768-dimensional vectors were searched with FAISS/HNSW. A separate cross-attention reader remained extractive. Single-dataset DPR versus BM25 top-20 answer-containing recall: | Dataset | DPR | BM25 | |---|---:|---:| | Natural Questions | 78.4 | 59.1 | | TriviaQA | 79.4 | 66.9 | | WebQuestions | 73.2 | 55.0 | | CuratedTREC | 79.8 | 70.9 | | SQuAD | 63.2 | 68.8 | End-to-end EM was 41.5, 56.8, 34.6, 25.9, and 29.8 respectively. The paper's 995-query/s DPR versus 23.7-query/s-per-Lucene-thread comparison was specific to its 512GB CPU setup; passage encoding and HNSW construction each took roughly 8.5–8.8 hours while Lucene indexing took about 30 minutes. Supervised/weak positives, answer-string false positives, full corpus re-encoding after encoder changes, and rare-term errors remain. DPR's minimal bi-encoder + in-batch negatives + lexical hard negative + FAISS recipe became the default retriever beneath RAG and FiD. [EMNLP paper](https://aclanthology.org/2020.emnlp-main.550/). --- ## 4. The 2020 generative convergence ### RAG: latent documents plus pretrained seq2seq generation Patrick Lewis and colleagues coupled DPR with BART-large. The retriever gives \[ p_\eta(z\mid x)\propto\exp(d(z)^\top q(x)). \] RAG-Sequence uses one latent passage for the entire output: \[ p(y\mid x)\approx\sum_{z\in\operatorname{top}k} p_\eta(z\mid x)\prod_i p_\theta(y_i\mid x,z,y_{