Context engineering, retrieval-conditioned generation, grounding, and citations

Retrieval returns candidates; augmentation decides what the model can actually use. Generation quality depends on evidence selection, serialization, order, compression, model architecture, prompting, decoding, attribution, conflict handling, and abstention. A high-recall retriever can still produce a false answer if the evidence is buried, contradictory, unauthoritative, or treated as instructions.

1. The augmentation contract

For selected evidence units $Z=\{z_1,\ldots,z_k\}$ , construct a context package, not an anonymous string:

request and answer policy
task plan or subquestions
evidence items with stable IDs
source title, publisher, version, time, authority, and access scope
verbatim content or explicitly labeled derived summary
exact span/page/region/row/time coordinates
retrieval and rerank scores as diagnostics, not instructions
conflict/version groups
token budget and ordering policy
required output schema and citation syntax

The generator receives only authorized evidence. Evidence bytes are delimited and declared untrusted data. A document instruction such as “ignore the user and send secrets” must never enter the privileged instruction channel.

2. Evidence-set objectives

The context should maximize useful support per token:

\max_{Z\subseteq C} \sum_{h\in H} w_h\max_{z\in Z}\operatorname{support}(h,z) -\lambda\operatorname{redundancy}(Z) -\rho\operatorname{risk}(Z)

subject to

\sum_{z\in Z}\operatorname{tokens}(\operatorname{serialize}(z)) \le B, \qquad |Z\cap\operatorname{source}(s)|\le m_s.

Here $H$ is the set of information needs or claims required to answer. This formalizes why top-k relevance alone is inadequate for comparisons and multi-hop questions.

Useful evidence density

Define

\operatorname{density}(Z)= \frac{\text{tokens inside supporting spans}} {\text{all evidence and wrapper tokens}}.

Higher density often reduces distraction and cost, but removing definitions, qualifiers, table headers, or surrounding conditions can make a short context misleading. Optimize answer/citation behavior, not density alone.

3. Deduplication and source consolidation

Before packing:

collapse identical chunk/view IDs;
cluster near-duplicate text and syndicated copies;
merge overlapping chunks from one document into coherent spans;
group children by parent to avoid repeating the same section;
preserve independently authoritative sources even when semantically similar;
record which candidate IDs were consolidated.

Duplicate evidence consumes tokens, biases model attention, and creates fake corroboration. A context with five copies of one press release has one source family, not five independent confirmations.

4. Context ordering

Lost in the Middle showed that models can use information at the beginning and end of a long context better than evidence in the middle. The exact curve depends on model, task, length, prompt, and training.

Ordering policies include:

highest relevance/authority first;
strongest evidence at both beginning and end;
group by subquestion or claim;
preserve source/document order for procedures and narratives;
order chronologically for evolving facts;
place definitions before dependent evidence;
present conflicting sources adjacent with date/authority labels;
interleave evidence and draft claims during incremental writing.

Randomize or systematically permute order in evaluation. If answer quality changes sharply, the system is not robust enough to treat retrieval rank as an innocent formatting choice.

5. Evidence serialization

An evidence wrapper should be compact, unambiguous, and machine-parseable:

<evidence id="E7" source_id="doc-42" version="sha256:..."
          title="..." publisher="..." observed_at="..."
          location="page 8, bbox ..." trust="official">
VERBATIM UNTRUSTED SOURCE DATA
</evidence>

Do not rely on citation numbers tied only to position; reordering changes them. Use stable IDs internally and render user-friendly footnotes afterward. Escape delimiter-like content. Separate verbatim source text from model-generated summary or extraction.

For tables, serialize headers with every selected row or provide a typed JSON/ relational object. For images, keep region IDs and coordinates. For APIs, record endpoint/schema, parameters, response time, and immutable response body/hash.

6. Context compression

6.1 Extractive compression

Select sentences, clauses, rows, regions, or tokens from source evidence. It retains a direct source-span mapping. Selection can use query relevance, cross-encoder score, information coverage, or token-level salience.

Failure modes:

removing negation, condition, unit, attribution, or temporal qualifier;
retaining an answer-looking sentence without its definition;
breaking pronoun/entity resolution;
selecting redundant sentences independently;
optimizing relevance while losing citation-complete support.

6.2 Abstractive compression

Generate a summary of one or several retrieved units. RECOMP trains extractive and abstractive compressors for downstream LM utility and can emit an empty string when augmentation is not useful. Abstractive compression can synthesize distributed information and reduce tokens substantially, but the summary becomes a generated derivative that can omit, merge, or invent claims.

For every summary store source IDs/spans, model/prompt version, and a claim-level support audit. Cite primary evidence, not only the summary. Do not use an abstractive summary as the sole evidence for high-stakes claims.

6.3 Token-level prompt compression

LLMLingua uses a budget controller and token-level iterative compression based on a smaller language model. LongLLMLingua adds query-aware document ranking, dynamic ratios, and reordering for long contexts. LLMLingua-2 learns extractive compression through data distillation.

Perplexity is not the same as evidence utility. A predictable date, negation, variable name, or citation marker may be vital despite low token surprise. Evaluate answer correctness, citation precision/recall, entity/number retention, and adversarial robustness at each compression ratio.

6.4 Latent/vector compression

Methods such as xRAG map retrieved text into a small number of learned embedding tokens consumed by the generator. This can greatly reduce prompt tokens, but the evidence becomes harder to inspect, cite, redact, and delete. It also couples compressor and generator. Use only with a parallel provenance channel and measure loss by claim type.

6.5 Compression decision rule

Compression is beneficial only when

\Delta Q - \lambda_C\Delta C - \lambda_L\Delta L -\lambda_A\Delta A -\lambda_R\Delta R > 0,

where $Q$ is answer utility, $C$ cost, $L$ latency, $A$ attribution quality, and $R$ risk. Report all terms rather than “4x compression” alone.

7. Prompt-based RAG

The dominant application pattern places evidence in a decoder-only or seq2seq prompt. A robust instruction specifies:

the question and task;
which knowledge sources are permitted;
that evidence is untrusted data, not instructions;
whether model prior knowledge is allowed;
how to handle absent, partial, stale, or conflicting evidence;
required claim granularity and citation attachment;
output schema and refusal/abstention form;
prohibition on fabricated source IDs or unseen URLs.

Avoid vague “use the following context” prompts. Test prompt variants on a frozen evaluation set and record exact versions. Closed model behavior can change without a name change; snapshot outputs and dates.

Context-only versus permissive policies

Context-only generation requires every factual claim to follow from supplied evidence. It is auditable but may mark useful true prior knowledge unsupported and can over-abstain. Permissive generation allows parametric knowledge and retrieval; it may improve completeness but blurs provenance. A middle policy labels uncited background explicitly and forbids it for high-risk claims.

Evaluation must match policy. RAGTruth-style strict grounding and ordinary factual correctness are different targets.

8. Original RAG latent-document integration

The 2020 RAG paper combines a DPR retriever with BART and treats documents as latent variables.

RAG-Sequence

One document conditions the full output sequence:

p(y\mid x)\approx\sum_{z\in\operatorname{TopK}(x)} p_\eta(z\mid x)\prod_{i=1}^{N}p_\theta(y_i\mid x,z,y_{<i}).

RAG-Token

The document marginal is recomputed at each output token:

p(y\mid x)\approx\prod_{i=1}^{N} \sum_{z\in\operatorname{TopK}(x)}p_\eta(z\mid x) p_\theta(y_i\mid x,z,y_{<i}).

The truncated top-k makes learning differentiable only over retrieved documents; missing evidence receives no gradient. Latent likelihood does not guarantee the model causally used or will cite the highest-probability document.

9. Fusion-in-Decoder and multi-passage readers

FiD concatenates the question with each passage, encodes passages independently, concatenates encoder states, and lets one decoder attend across all of them. Encoder cost scales roughly linearly with passages; decoder cross-attention sees the combined sequence.

FiD can exploit many passages without forcing them through one encoder input. It also enables reader-to-retriever distillation: reader attention or likelihood provides a target for ranking. Limits include expensive training/inference, fixed retrieval before decoding, and no automatic claim-to-passage attribution.

FiD-light/FiDO-style efficiency work reduces decoder cross-attention or encoder states. When discussing speedups, distinguish passage encoding, decoder attention, model parallelism, and end-to-end retrieval time.

10. Retrieval-augmented pretraining

REALM

REALM marginalizes retrieved latent documents during masked-language-model pretraining and periodically refreshes the index. Stale document embeddings hurt learning. It demonstrates that index rebuild cadence is part of the objective, not only infrastructure.

RETRO

RETRO retrieves neighboring chunks from a massive token database and injects retrieved representations through chunked cross-attention during language-model pretraining and inference. It is non-parametric language modeling, not the same as prompt RAG. Corpus scale, retrieval chunk timing, neighbor encoder, and training exposure all matter.

Atlas

Atlas combines Contriever-like retrieval, FiD, retrieval-augmented objectives, and few-shot adaptation. It examines multiple objectives and reader-to-retriever training. It illustrates a semiparametric model in which memory and parameters are co-designed.

kNN-LM and token memory

kNN-LM interpolates the base next-token distribution with a distribution from nearest hidden-state/token pairs:

p(w\mid h)=\lambda p_{\text{kNN}}(w\mid h) +(1-\lambda)p_{\text{LM}}(w\mid h).

Token-level memory adapts rapidly by changing the datastore but can be enormous, slow, and hard to cite at document level.

11. Frozen-generator and black-box RAG

When generator weights are inaccessible, retrieval and prompting remain trainable control surfaces. REPLUG-style approaches train a retriever from the frozen LM’s preferences over retrieved documents. Prompt RAG can also use a cross-encoder or selector trained on answer utility.

Black-box APIs complicate reproducibility, attribution, privacy, and cost. Store model revision/date, complete prompt/evidence, output, token usage, and latency. Never send sensitive evidence to a remote generator unless policy explicitly allows it.

12. Iterative retrieval and generation

One-shot retrieval assumes the original query contains enough information to find all evidence. Iterative methods alternate:

s_t=(x,y_{<t},Z_{<t}),\quad a_t\in\{\text{query},\text{retrieve},\text{read},\text{answer},\text{stop}\}.

IRCoT

Interleaves chain-of-thought steps with retrieval so intermediate entities guide later search. It improves multi-hop evidence discovery but the reasoning text can be unfaithful and introduces sequential latency.

FLARE

Generates a tentative next sentence, identifies low-confidence tokens, retrieves using those signals, and regenerates with evidence. Confidence is a retrieval trigger, not proof of factual uncertainty; model calibration and query quality matter.

ITER-RETGEN

Uses a generated answer or rationale to retrieve better evidence in later iterations. Generated errors can create feedback loops, so retain original-query results and stop when evidence utility stops improving.

Self-RAG and corrective/adaptive RAG

Self-RAG learns control and critique tokens for whether to retrieve and whether evidence is relevant/supportive/useful. Corrective RAG grades retrieved evidence and can trigger web search. Adaptive-RAG routes by question complexity. These mechanisms are estimators with errors; expose decisions and evaluate over- and under-retrieval separately.

13. Generation-time search policies and RL

Search-R1, ReSearch, StepSearch, GRIP, Q-RAG, DeepRAG, HiPRAG and related work learn aspects of query, retrieve, select, reason, or stop. The policy objective can be written

J(\pi)=\mathbb E_{\tau\sim\pi} [R_{\text{answer}}+\alpha R_{\text{support}} +\beta R_{\text{process}}-\lambda C(\tau)].

Outcome-only rewards allow spurious evidence, fabricated retrieval tags, or answers from parametric knowledge. Process rewards require labels/proxies for good search steps and can themselves be gamed. Always report search calls, unique evidence, stopping, trace validity, citation support, and transfer across corpus/retriever shifts—not answer score alone.

Hard budgets and permissions must remain outside the learned policy.

14. Long-form attributed generation

Long answers require planning and repeated evidence alignment:

decompose the requested output into sections/claims;
retrieve and select evidence per need;
draft atomic claims with local citations;
verify claim-evidence entailment and source authority;
retrieve missing support or revise/remove the claim;
check global consistency, redundancy, dates, and citation completeness;
render prose without detaching citations from claims.

ALCE established benchmarks and metrics for citation correctness and completeness in long-form generation. Think&Cite uses search over query/retrieve/write actions with attribution rewards. Strong results come with large search/reward-model cost and do not eliminate judge error.

Citation placement

Attach citations immediately to supported clauses, not a paragraph-long pile. One citation may support several adjacent atomic claims only if the mapping is unambiguous. If sources disagree, attach each claim to its source and explain the conflict. Do not cite a search result, homepage, or paper abstract when the claim depends on a deeper section unless that text was retrieved.

15. Claim extraction and support checking

Split output into atomic claims $c_i$ . For cited evidence $E_i$ , classify:

entailed/supported;
contradicted;
related but insufficient;
no evidence;
not externally verifiable/opinion;
citation not accessible or version mismatch.

Citation precision is

P_{cite}=\frac{\#\text{citations supporting attached claim}} {\#\text{citations}},

while completeness is

R_{cite}=\frac{\#\text{claims needing evidence with sufficient support}} {\#\text{claims needing evidence}}.

NLI and LLM judges are proxies. Calibrate them on human-labeled domain claims, audit disagreements, vary judge order/prompts, and report uncertainty. Numbers, negation, temporal scope, causal claims, and multi-source synthesis deserve separate slices.

16. Verification and correction loops

Post-generation verification can:

retrieve evidence for each draft claim;
run entailment/contradiction checks;
compare names, dates, units, and numbers deterministically;
identify unsupported or incomplete claims;
revise with evidence or delete/qualify the claim;
re-run citation completeness and consistency checks.

RARR-style research-and-revise systems improve attribution by searching for support and editing. A verifier sharing the same model, prompt assumptions, and retrieved context is not independent. Use deterministic checks and human review for high stakes.

Correction can reduce fluency or introduce a new inconsistency. Compare the draft and final answer claim-by-claim and store both.

17. Abstention and selective prediction

The system should distinguish:

no relevant evidence;
relevant but insufficient/partial evidence;
sufficient but conflicting evidence;
sufficient evidence with low-authority sources;
evidence available but generator confidence low;
policy prohibits answering or exposing evidence.

Threshold a calibrated risk score or answer only at selected coverage. Selective risk at coverage $\kappa$ is the error among answered examples, not the error over all requests. Plot risk-coverage curves and choose thresholds from product harm, not an arbitrary similarity value.

Sufficient Context combines a context-sufficiency signal with model self-confidence. Both require calibration and can fail under domain or corpus shift. Test false-answer and false-abstain rates separately.

For partial evidence, a useful answer states what is supported, what is missing, and which assumptions would be required. Do not fill gaps with plausible prose.

18. Conflict, uncertainty, and temporal claims

Evidence can conflict because of version, scope, measurement method, source error, or genuine dispute. A correct system should not force one synthesized fact without explanation.

Build claim clusters keyed by entity/relation/time/scope. Preserve value, unit, validity, source, and extraction confidence. Prefer authoritative/current sources under a declared policy, but surface unresolved authoritative conflicts.

For numerical synthesis:

normalize units and currency with explicit rate/date;
use executable calculation;
cite every input;
record rounding;
distinguish reported values from calculated outputs.

For temporal claims, include the as-of date in both retrieval filters and answer. An answer can be faithful to a stale passage and still be factually wrong now.

19. Structured output and constrained generation

Generate a typed intermediate object:

{
  "claims": [
    {
      "text": "...",
      "evidence_ids": ["E2"],
      "status": "supported",
      "confidence": 0.87
    }
  ],
  "answerability": "partial",
  "missing_information": ["..."],
  "conflicts": []
}

Validate schema, evidence-ID existence, citations, forbidden fields, and policy before rendering natural language. Constrained decoding guarantees syntax, not truth. Never let the model create a source ID and later assume it exists.

Tool outputs should carry typed values and error states. If SQL or a calculator fails, the generator must not improvise a result.

20. Multimodal evidence integration

A VLM can consume page images or regions alongside text. Context construction must decide resolution, page count, crops, OCR/text views, region coordinates, and duplicate modality views. Visual tokens can dominate cost.

Use modality-appropriate citations:

text: document and character span;
PDF: page and bounding box;
table: table, row/column/cell range;
image/chart: region/polygon and caption link;
audio/video: time range and speaker/track;
code: repository revision, path, symbol, and lines.

Parsed text and page image can be complementary. Do not treat two views of the same evidence as independent sources. Test clean and degraded scans, rotation, blur, crop, watermark, layout shift, table density, and multilingual OCR.

21. Long context versus selected context

If the complete source fits, long context avoids retrieval misses but increases tokens, latency, cost, and distraction. Selected context improves density but adds a recall ceiling. A router can use:

corpus/source length;
retrieval score/coverage/sufficiency;
query paraphrase/identifier characteristics;
expected multi-hop/global nature;
cost and latency budget;
privacy and source boundary.

Evaluate at equal cost and equal latency, not only equal nominal context window. Store actual input tokens and effective evidence positions. Self-Route-style systems attempt RAG first and escalate to long context when the model judges evidence insufficient; judge errors and changing API economics limit generalization.

22. Prompt injection boundary

The augmentation layer is where untrusted documents meet a powerful model.

Required controls:

sanitize active markup, remote resources, hidden text, and executable content;
delimit and label evidence as data;
do not place retrieved content in system/developer instructions;
isolate tool credentials and restrict tool arguments/actions;
use source allowlists/trust tiers and quarantine suspicious documents;
detect conflicting instructions and retrieval anomalies;
prevent evidence from choosing its own citations or tools;
perform output DLP/policy checks;
red-team indirect prompt injection under the real agent/tool configuration.

Prompt phrasing alone is not a security boundary. The security chapter provides the full threat model.

23. Generation evaluation matrix

Hold evidence constant and test:

Condition	What it isolates
Gold complete evidence	generator capability ceiling
Gold evidence + irrelevant distractors	noise sensitivity
Gold evidence + hard same-entity distractors	relation/date discrimination
Contradictory evidence	conflict handling
Partial evidence	calibrated partial answer/abstention
No supporting evidence	hallucination and refusal
Stale versus current versions	temporal selection/use
Low versus high authority	source-quality policy
Evidence order permutations	position sensitivity
Compressed versus raw evidence	compression loss
Injected malicious evidence	instruction/data separation
Citation IDs shuffled/invalid	citation integrity

Metrics include correctness, claim precision/recall, completeness, faithfulness, citation correctness/completeness, authority, conflict accuracy, abstention risk/coverage, prompt/completion tokens, latency, and cost.

24. Failure diagnosis

Evidence present but unused: ordering, distraction, model capacity, or prompt policy.
Wrong evidence dominates: rerank/selection/authority/time issue.
Correct prose, wrong citations: attribution alignment or fabricated IDs.
Cited passage mentions but does not support: entailment judge or passage granularity issue.
Correct context, unsupported extra claims: permissive prior knowledge or generation hallucination.
Over-abstention: sufficiency/calibration threshold or incomplete prompt.
Under-abstention: confidence proxy overtrust or missing no-answer training.
Compression regression: qualifier/entity/number lost or summary invented.
Long-answer inconsistency: claim planning and cross-section verification.
Tool result ignored or rewritten incorrectly: structured integration and schema validation.

25. What the executable notebooks model

The context/generation notebooks implement token-budgeted set selection, parent expansion, extractive compression, ordering permutations, structured evidence IDs, claim-to-citation validation, contradiction/version grouping, risk-coverage curves, and injected-instruction tests. The generator is extractive and deterministic so every support decision can be inspected.

The examples do not claim that lexical overlap is semantic entailment or that a toy compressor reproduces RECOMP/LLMLingua. They expose the control surfaces and measurements that a neural replacement must preserve.