The Answer Must Touch the Evidence

When retrieved evidence enters a language model, what makes the resulting answer grounded rather than merely accompanied?

Field notes on retrieval-conditioned generation, context, citation, and the right to remain silent

Retrieval makes a promise that generation can easily break. A passage may be found, ranked correctly, cleared for access, and placed in the prompt; the model may still ignore its qualifier, combine it with an incompatible source, or answer from parametric memory and decorate the result with a nearby citation. The presence of evidence is not yet grounding.

The boundary between retriever and generator is therefore not a text concatenation. It is a contract. The generator should receive a package that states the request, the answer policy, the selected evidence and its stable identities, the source versions and locations, any conflicts, and the required form of attribution. Retrieved bytes are untrusted data, not instructions. A score is diagnostic metadata, not a command to believe.

The governing objective is not to fill a context window. It is to maximize useful support per token while preserving the conditions that make the support valid:

\max_{Z\subseteq C} \sum_{h\in H}w_h\max_{z\in Z}\operatorname{support}(h,z) -\lambda\operatorname{redundancy}(Z) -\rho\operatorname{risk}(Z),

subject to a token budget on the serialized evidence. Here $H$ is the set of claims or information needs the answer must satisfy. This formulation explains why generation begins with selection. A pile of highly relevant passages may repeat one fact and omit the second half of a comparison.

1. Retrieval enters the learning objective

Neural systems retrieved before the name RAG existed. Memory Networks read external slots; DrQA retrieved Wikipedia before extracting spans; 2018's Retrieve and Refine conditioned a dialogue model on a retrieved response; Wizard of Wikipedia paired conversation with selected knowledge. The 2020 convergence was more specific: pretrained language models, large external indexes, and output likelihood were joined in one trainable probabilistic account.

REALM made retrieval part of masked-language-model pretraining. It treated a document $z$ as latent:

p(y\mid x)=\sum_{z\in\mathcal Z}p(y\mid x,z)p(z\mid x),\qquad p(z\mid x)\propto\exp(E_x(x)^\top E_z(z)).

A BERT bi-encoder proposed evidence; another BERT cross-encoded the input and document. A document received positive signal when it increased the likelihood of the masked target. Salient entity and date masking made retrieval useful, while a null document, exclusion of the source document, and Inverse Cloze initialization discouraged easy shortcuts.

REALM used just over 13 million blocks from a December 2018 English Wikipedia snapshot. It marginalized eight candidates during pretraining and five during open-domain QA. Cached document embeddings were rebuilt asynchronously about every 500 steps; at downstream QA time, the document encoder and index were frozen. This detail is not incidental infrastructure. In the paper, making the index thirty times staler reduced Natural Questions development exact match from 38.2 to 28.7. The memory and the learner must agree about the representation space.

The system remained extractive and expensive—the reported pretraining used 64 TPUs—but it established a durable principle: retrieved documents can be latent causes inside the training objective rather than static features appended after training.

2. RAG gives the convergence a name

The original Retrieval-Augmented Generation paper coupled a DPR retriever with BART-large. The retriever assigned

p_\eta(z\mid x)\propto\exp(d(z)^\top q(x)),

and the generator conditioned on both input and passage. RAG-Sequence chose one latent passage for the full output:

p(y\mid x)\approx \sum_{z\in\operatorname{TopK}(x)} p_\eta(z\mid x)\prod_i p_\theta(y_i\mid x,z,y_{<i}),

whereas RAG-Token moved the document marginal inside the product:

p(y\mid x)\approx \prod_i\sum_{z\in\operatorname{TopK}(x)} p_\eta(z\mid x)p_\theta(y_i\mid x,z,y_{<i}).

The distinction is almost architectural handwriting. RAG-Sequence asks one passage to sustain a sequence; RAG-Token permits the evidence mixture to change as each token is written. Both optimize negative marginal log likelihood over the retrieved top set. Neither can send learning signal to a relevant passage that failed to enter that set.

The implementation fine-tuned BART and the question encoder while freezing the document encoder and its 21-million-passage Wikipedia index. The retriever also inherited DPR supervision from Natural Questions and TriviaQA. RAG was thus not a retrieval-label-free system, despite the latent-document objective.

The results justified the new formulation without proving automatic grounding. RAG-Sequence reached test exact match of 44.5 on Natural Questions and 45.2 on WebQuestions, compared with cited DPR results of 41.5 and 41.1. On MS MARCO it moved ROUGE-L from BART's 38.2 to 40.8. Human raters comparing Jeopardy-style generations preferred RAG's factuality in 42.7 percent of pairs and BART's in 7.1 percent.

Yet on Natural Questions, RAG still answered 11.8 percent correctly when no retrieved passage contained the answer. Parametric memory could rescue a retrieval failure, but the same freedom could also override evidence. The model produced no guarantee that an output was entailed by a passage, offered no abstention mechanism, and supplied no claim-level citation. Marginal likelihood is a learning signal, not an audit trail.

The paper's index-swapping experiment showed the other side of external memory. Models paired with matched 2016 or 2018 indexes answered corresponding world-leader probes at 70 and 68 percent; mismatched indexes fell to 12 and 4 percent. Updating an index can move factual behavior without retraining the generator. It can also create temporal incoherence if the index, prompt cache, and citations refer to different snapshots.

3. FiD lets passages remain themselves

Early fusion concatenates all retrieved text into one encoder input and pays quadratic self-attention across the whole sequence. It also invites passages to blur before the decoder sees them. Fusion-in-Decoder took a cleaner route: concatenate the question with each title and passage, encode every pair independently with T5, join the resulting encoder states, and allow one decoder to attend over the union.

question + passage 1  --> encoder --\
question + passage 2  --> encoder ---+--> decoder --> answer
question + passage n  --> encoder --/

Encoder self-attention now scales roughly linearly with passage count because passages do not attend to one another there. The decoder performs the fusion. FiD-base and FiD-large normally used 100 passages truncated to 250 wordpieces. The paper reported Natural Questions test exact match of 48.2 and 51.4, respectively; FiD-large reached 67.6 on open TriviaQA. Increasing from ten to one hundred passages improved Natural Questions development exact match by 3.5 points and TriviaQA by about six.

The architecture created a strong multi-passage reader and a useful teacher. Later FiD-KD work aggregated decoder cross-attention into passage preferences and distilled them into a dual encoder. But attention remains a heuristic for importance, not causal proof, and FiD itself leaves retrieval fixed before decoding. Its large encoder-state bundle is costly, and nothing in the architecture automatically maps each generated claim back to a passage.

This distinction matters: evidence integration and attribution are separate design problems. A model may synthesize well across one hundred passages and still be unable to show which sentence supports which clause.

4. Memory scales in another direction

kNN-LM demonstrated that external memory could intervene at every next-token decision without retraining the base model. It stored hidden-state contexts as keys and their following tokens as values, then interpolated a nearest-neighbor distribution with the language model:

p(w\mid h)=\lambda p_{\mathrm{kNN}}(w\mid h) +(1-\lambda)p_{\mathrm{LM}}(w\mid h).

On WikiText-103, the reported test perplexity moved from 18.65 to 16.12. A datastore could also be swapped to adapt domains. The cost was an entry and lookup per token, immense storage, and evidence whose document provenance was awkward to expose.

RETRO moved this non-parametric language-model idea from individual token states to chunks. It divided each 2,048-token training sequence into 64-token chunks. A frozen BERT embedding and ScaNN retrieved a neighbor chunk plus its following 64-token continuation. A bidirectional neighbor encoder and Chunked Cross-Attention injected that 128-token value while preserving causal generation: a previous chunk's retrieval informs current predictions.

The database scale was the point. MassiveText contained more than five trillion raw tokens; ordinary training retrieval used 600 billion, while evaluation used a 1.75-trillion-token index, rounded in the paper's “two trillion” framing. The MassiveText index occupied 93 TB. RETRO-7.5B was comparable to substantially larger GPT-3 and Jurassic models on many, not all, Pile subsets. Its Natural Questions test exact match was 45.5 with DPR passages, between RAG's 44.5 and FiD's 51.4 in the paper's comparison.

RETRO showed that parameter count and memory size could scale separately. It also exposed the hazards at that scale: a frozen similarity model, proprietary data, copying and privacy questions, enormous storage, and contamination. WikiText-103 perplexity of 3.92 with the 1.8-trillion-token datastore was explicitly partly due to leakage. Retrieval-augmented pretraining does not make the provenance problem disappear; it can make the provenance surface enormous.

5. Atlas co-designs retriever and reader

Atlas brought several lines together: an unsupervised Contriever-style retriever, T5 models at 770 million, 3 billion, and 11 billion parameters, FiD integration, retrieval-augmented pretraining, and few-shot adaptation. It compared four retriever objectives. The selected likelihood-distillation target was

p_{\mathrm{LDist}}(d_k)\propto p_{\mathrm{LM}}(a\mid d_k,q),

with a KL objective transferring the reader's preference among documents into the retriever. Masked language modeling with 15 percent masking and mean span length three was the selected pretraining task.

Atlas indexed a December 2021 Wikipedia, including linearized lists and infoboxes, as 37 million section passages, alongside roughly 350 million CCNet passages. Pretraining retrieved 100 candidates from a stale index, re-embedded and reranked them to 20, and refreshed the index every 2,500 steps. Downstream query-side tuning avoided a full reindex.

The 11-billion-parameter model reached 42.4 Natural Questions exact match with 64 examples and 60.4 with full data using the mixed index. A temporally matched 2018 Wikipedia raised those figures to 45.1 and 64.0. A TempLAMA-derived experiment made the index's temporal agency vivid: a 2017 model and index scored 57.7 on 2017 facts and 1.5 on 2020 facts; swapping only to a 2020 index changed the scores to 10.2 and 53.1.

The often repeated comparison between Atlas-11B at 42.4 and PaLM-540B at 39.6 must retain its experimental grammar: the former used 64-example fine-tuning, the latter prompting. It is evidence of sample efficiency, not a controlled architecture duel. Atlas's larger contribution was the co-design itself: pretraining, retrieval, reader preference, and replaceable memory treated as one semiparametric system.

6. Context is an arrangement, not a quantity

Once passages are selected, three mundane operations govern whether the model can use them: consolidation, ordering, and serialization.

Duplicate chunks should be collapsed by stable identity; overlapping children from one document should be merged; syndicated copies should not masquerade as independent corroboration. Five versions of one press release consume tokens and bias attention, but still constitute one source family. Independently authoritative sources should remain separate even when their wording is similar.

Ordering is consequential. Lost in the Middle showed that language models can use relevant information at the beginning and end of long contexts better than information placed in the middle, with the exact curve dependent on model, task, prompt, and length. Retrieval rank is therefore not innocent formatting. Definitions may need to precede dependent facts; procedures should retain source order; changing claims should be ordered temporally; conflicting sources should be adjacent and labeled.

Serialization should preserve stable evidence IDs rather than footnote numbers derived from prompt position:

<evidence id="E7" source_id="doc-42" version="sha256:..."
          title="..." observed_at="..." location="page 8">
VERBATIM UNTRUSTED SOURCE DATA
</evidence>

Tables need headers attached to selected rows. Images need regions and coordinates. API evidence needs request parameters, response time, and a retained body or hash. Generated summaries must be labeled as derivatives and retain links to their source spans.

Compression sharpens the trade. Extractive compression preserves a span map, but may delete a negation, unit, attribution, or governing condition. Abstractive compression can combine dispersed evidence, but creates another generated object that can omit or invent. Compression is successful only when it preserves answer quality and citation behavior, not when it merely reports a large token ratio.

Long context offers no automatic escape. Supplying every source avoids a retrieval miss only when the complete, authorized corpus fits, and it increases cost, latency, distraction, and positional sensitivity. Selected context adds a recall ceiling but can improve evidence density. Compare the two at equal cost or latency, record actual token positions, and route according to source length, query type, sufficiency, and risk.

7. Citation is a claim-level relation

A grounded answer is not prose followed by a bibliography. It is a set of atomic claims, each connected to the evidence that entails it. For cited evidence $E_i$ , a verifier should distinguish supported, contradicted, merely related, absent, inaccessible, and version-mismatched cases.

Citation precision asks what fraction of citations support their attached claim; citation completeness asks what fraction of externally verifiable claims have sufficient support:

P_{\mathrm{cite}}= \frac{\#\text{supporting attached citations}}{\#\text{citations}}, \qquad R_{\mathrm{cite}}= \frac{\#\text{supported claims needing evidence}} {\#\text{claims needing evidence}}.

Neither measure captures everything. Source authority, provenance validity, and whether the evidence causally influenced generation remain separate. A hyperlink may sit beside a true statement while pointing to an irrelevant page; a malicious page may perfectly entail a claim; a source appended after drafting may create the appearance of grounding.

KILT made provenance over a fixed Wikipedia snapshot part of knowledge-intensive evaluation. ALCE later supplied benchmarks and metrics for citation correctness and completeness in long-form answers. Both reinforce a design rule: attach citations immediately to the clauses they support, with immutable internal IDs resolved to exact spans, pages, rows, or regions. Do not cite a search result or abstract when the claim depends on text deeper in the source.

An answer pipeline can draft claims, validate every referenced ID, run entailment and contradiction checks, compare names and numbers deterministically, then revise or delete unsupported material. The draft and the corrected answer should both be retained. A verifier built from the same model and assumptions is not independent merely because it runs second.

8. Abstention completes the architecture

The final retrieval decision is sometimes not to answer. “No evidence” is only one state. Evidence may be relevant but incomplete, sufficient but conflicting, low-authority, stale, inaccessible under policy, or adequate while the generator remains uncertain.

These states should not be collapsed into a single similarity threshold. Retrieval score is not answer probability, and generation confidence is not context sufficiency. The Sufficient Context study combines a context-sufficiency signal with model self-confidence; both still require calibration and can drift with corpus and domain.

Selective prediction makes the trade visible. At coverage $\kappa$ , measure risk among the examples the system chose to answer, then plot risk against coverage. A model can look more accurate by refusing every difficult request. Report false answers and unnecessary abstentions separately, and choose the operating point from product harm rather than an arbitrary vector score.

For partial evidence, the best response often has three parts: what the sources support, what is missing, and which assumptions would be required to continue. That is more useful than either fluent invention or a featureless refusal.

Internally, the answer can be represented before it becomes prose:

question
   |
   v
selected evidence --> atomic claims --> support audit
                           |                 |
                           +-------> answer / qualify / abstain

A typed record might carry the claim text, evidence IDs, support status, confidence, missing information, and conflicts. Schema constraints guarantee only structure, not truth, but they make invalid IDs and unsupported claims detectable before rendering.

The mature retrieval-augmented system is therefore not a generator with a search call in front of it. It is a versioned evidence process that can show how an answer was assembled, tell when that assembly is incomplete, and decline to hide the gap with language. Retrieval gives the model somewhere else to look. Grounding requires it to keep looking back.