# The complete RAG field map

This chapter defines the coverage contract for the repository. It treats
retrieval-augmented generation (RAG) as a complete information system rather
than the three-box diagram `retrieve -> concatenate -> generate`. A system is
not specified until its corpus, representation, index, query policy, evidence
selection, generator, attribution policy, evaluation protocol, security model,
and operating envelope are specified.

**Evidence cutoff:** 2026-08-09. “Complete” below means that every established
RAG subsystem and major research family is named, related to adjacent families,
and routed to a detailed chapter. It does not mean that every paper ever using
the acronym RAG is individually summarized. Paper-level claims use primary
sources and retain their publication status.

## 1. What belongs inside RAG

For a request \(x\), tenant and policy state \(a\), corpus snapshot
\(\mathcal C_t\), retrieved evidence \(Z\), and answer \(y\), an operational
RAG system is a composition

\[
\mathcal C_t
\xrightarrow{\text{parse, normalize, segment, enrich}}
\mathcal U_t
\xrightarrow{\text{represent, index}}
\mathcal I_t,
\]

\[
(x,a)
\xrightarrow{\pi_{\text{plan}}}
q_{1:m}
\xrightarrow{\pi_{\text{retrieve}}(\mathcal I_t)}
R_{1:m}
\xrightarrow{\pi_{\text{select}}}
Z
\xrightarrow{p_\theta(y\mid x,Z,a)}
(y,A,c),
\]

where \(\mathcal U_t\) is the set of retrievable units, \(\mathcal I_t\) is one
or more indexes, \(q_{1:m}\) are search actions, \(R_{1:m}\) are candidate
lists, \(A\) is claim-to-source attribution, and \(c\) is calibrated confidence
or abstention state. Every arrow can be fixed, learned, or hybrid.

RAG includes all of the following:

1. **Knowledge acquisition:** connectors, crawling, change-data capture,
   snapshots, permissions, deletion, and licensing.
2. **Document intelligence:** file decoding, OCR, layout, reading order, table,
   chart, formula, image, audio, code, and structured-record extraction.
3. **Corpus engineering:** normalization, deduplication, language detection,
   metadata, versioning, entity linking, and quality/trust scoring.
4. **Retrieval-unit construction:** documents, passages, sentences,
   propositions, windows, parent-child units, summaries, graph nodes, rows,
   regions, image patches, and learned memory vectors.
5. **Representation:** lexical postings, learned sparse weights, single dense
   vectors, multi-vectors, cross-modal vectors, symbolic triples, and hybrid
   representations.
6. **Indexing:** inverted files, exact matrix search, locality-sensitive
   hashing, trees, product quantization, IVF, navigable graphs, disk indexes,
   graph stores, SQL, and federated search.
7. **Query understanding:** intent, entities, filters, time, geography,
   permissions, conversation state, ambiguity, answerability, and complexity.
8. **Query transformation:** correction, expansion, pseudo-relevance feedback,
   hypothetical documents, rewriting, decomposition, step-back questions,
   multilingual translation, and tool/API plans.
9. **Candidate generation:** sparse, dense, late-interaction, graph, table,
   multimodal, web, API, memory, and ensemble retrieval.
10. **Selection:** fusion, reranking, filtering, deduplication, diversity,
    coverage, contradiction handling, trust, freshness, and token budgeting.
11. **Evidence integration:** prompt concatenation, FiD-style fusion, latent
    marginalization, cross-attention, retrieval during decoding, and
    compressed/structured evidence.
12. **Generation control:** retrieve/no-retrieve routing, iterative search,
    planning, stopping, verification, correction, citation, and abstention.
13. **Learning:** retriever contrastive learning, hard-negative mining,
    reader-to-retriever distillation, joint latent training, supervised search
    trajectories, preference optimization, outcome RL, and process rewards.
14. **Evaluation:** retrieval, selection, generation, attribution, calibration,
    robustness, freshness, security, latency, cost, and human utility.
15. **Operations:** serving, caching, index migration, observability, incident
    response, capacity, SLOs, privacy, governance, and deletion guarantees.

The term also overlaps with non-parametric language modeling, tool-using
agents, search-augmented reasoning, long-term memory, in-context learning, and
knowledge-graph QA. This handbook includes them when an external store is
addressed at inference or training time and the retrieved state changes model
behavior. It does not call ordinary fine-tuning, a longer static prompt, or a
database lookup that never influences generation “RAG.”

## 2. The fourteen design axes

Two systems both called RAG may share almost no implementation. The following
axes make comparisons precise.

| Axis | Main choices | What can fail |
|---|---|---|
| Knowledge boundary | public web, curated corpus, tenant data, user memory, APIs, KGs | incomplete, unlawful, stale, or unauthorized content |
| Update semantics | immutable snapshot, append-only, CDC, bitemporal versions, live search | index lag, cache staleness, historical facts overwritten |
| Unit | document, section, passage, sentence, proposition, row, region, node | boundary cuts evidence or creates too much noise |
| Representation | sparse, learned sparse, dense, multi-vector, graph, visual, hybrid | vocabulary gap, embedding collision, extraction loss |
| Index | exact, inverted, IVF/PQ, HNSW, DiskANN, graph/SQL | approximation loss, memory pressure, filter failure |
| Query policy | fixed, rewritten, decomposed, iterative, learned | intent drift, query explosion, reward hacking |
| Candidate policy | one retriever, hybrid, routed, federated | recall ceiling, duplicate candidates, domain mismatch |
| Selector | pointwise, pairwise, listwise, setwise, MMR, trust-aware | relevant evidence removed or distractors retained |
| Integration | concatenate, marginalize, FiD, recurrent retrieval, latent vector | position bias, lost provenance, token cost |
| Generator | extractive, seq2seq, decoder-only, VLM, tool agent | unsupported synthesis, instruction following from data |
| Grounding policy | permissive prior knowledge, context-only, cited, abstaining | true-but-uncited or cited-but-unsupported claims |
| Learning signal | labels, weak labels, synthetic data, distillation, RL | false negatives, spurious evidence, judge/reward bias |
| Trust boundary | single tenant, ACL-filtered multitenant, untrusted web | leakage, poisoning, prompt injection, source spoofing |
| Objective | accuracy, coverage, faithfulness, latency, cost, risk | optimizing one proxy degrades the real product goal |

This is why “Which vector database or embedding model is best?” is not a
well-posed RAG question. The correct unit of analysis is a versioned pipeline
evaluated on a declared workload and threat model.

## 3. Knowledge-source taxonomy

### 3.1 Unstructured text

Examples include web pages, manuals, tickets, email, contracts, scientific
papers, and transcripts. The apparent simplicity is deceptive: navigation,
boilerplate, footnotes, headers, lists, references, code blocks, and section
hierarchy all affect retrieval. Text RAG should preserve stable source IDs,
character offsets, headings, and version timestamps rather than retaining only
plain chunk strings.

### 3.2 Visually rich documents

PDF is a presentation format, not a semantic document model. The pipeline must
recover reading order, columns, captions, tables, formulas, figures, and page
coordinates. OCR character error rate alone is insufficient because a small
structural error can destroy downstream retrieval. Visual-page retrieval such
as [ColPali](https://proceedings.iclr.cc/paper_files/paper/2025/hash/99e9cf99cc114c46c2e6168e4dc0c43a-Abstract-Conference.html)
avoids some parser loss by indexing image patches, but increases index size and
does not by itself provide answer-level attribution.

### 3.3 Tables and databases

Flattening a table loses row/column identity, headers, types, units, and join
structure. Options include row or cell retrieval, schema/table retrieval plus
SQL, table-to-text serialization, table graphs, and hybrid text-table plans.
[T2-RAGBench](https://aclanthology.org/2026.eacl-long.8/) and
[T-RAG](https://aclanthology.org/2026.findings-acl.1902/) demonstrate that
retrieving the correct table and performing numerical or cross-table reasoning
must be evaluated separately.

### 3.4 Knowledge graphs

Graphs may be curated KGs, LLM-extracted entity/relation graphs, document-link
graphs, passage/entity bipartite graphs, temporal event graphs, or a temporary
query-specific graph. Retrieval can use entity linking, neighborhood expansion,
path search, Personalized PageRank, communities, graph neural networks, or
LLM-guided traversal. Graph construction quality is often the recall ceiling;
graph RAG is not universally better than text RAG.

### 3.5 Images, charts, video, and audio

Systems can retrieve captions/OCR, global modality embeddings, page/image
patches, regions, frames, segments, or multimodal graph nodes. Alignment between
query modality, evidence modality, and generator matters: a text retriever can
never recover a visual relation omitted by its caption. Temporal media also
requires segment boundaries and synchronization.

### 3.6 Source code and repositories

The natural unit may be a symbol, function, class, call path, diff, issue, test,
or dependency—not a fixed token window. Lexical identifiers, AST structure,
repository topology, build configuration, and revision state are complementary
signals. Generated code needs provenance, license awareness, and execution or
test validation in addition to text similarity.

### 3.7 Conversation and long-term memory

Memory RAG distinguishes raw turns, extracted facts, preferences, summaries,
episodes, and latent states. It needs write, consolidate, update, supersede,
forget, and delete policies. Retrieval accuracy is only one part of correct
memory behavior; temporal ordering and conflict resolution are essential.

### 3.8 Live tools and APIs

Search engines, calculators, databases, weather/finance endpoints, and internal
services return dynamic evidence. Tool schemas, authentication, rate limits,
timeouts, idempotency, and observed timestamps become part of the RAG trace.
Live retrieval improves potential freshness but makes reproducibility require a
stored response snapshot.

### 3.9 Model-internal non-parametric memory

kNN-LM, RETRO, Memorizing Transformers, and latent-memory systems retrieve
token-level or vector-level neighbors inside model computation. They share the
external-memory principle but differ from prompt RAG in training integration,
granularity, index scale, and auditability.

### 3.10 Federated and permissioned sources

Enterprise queries may span stores with different owners, schemas, regions,
classifications, and retention policies. Retrieval must enforce authorization
before candidate exposure, preserve policy metadata through reranking, and
merge rankings without leaking inaccessible document existence.

## 4. Query and task taxonomy

RAG design starts from information need, not component fashion.

| Task | Evidence shape | Appropriate starting pattern | Typical hidden failure |
|---|---|---|---|
| Exact lookup | one passage, rare name/code/date | BM25 or learned sparse + metadata filters | dense embedding misses identifier |
| Paraphrastic lookup | one semantically matching passage | dense + sparse hybrid | lexical-only recall failure |
| Comparison | facts about two or more entities | decomposition + set coverage + cited synthesis | one side missing but answer still fluent |
| Multi-hop | linked facts across sources | iterative, graph, or decomposed retrieval | independently relevant chunks do not form a chain |
| Aggregation | many records or documents | structured query/map-reduce | sampling top-k biases aggregate |
| Global synthesis | themes over a corpus | hierarchical summaries/communities | local chunks omit corpus-level distribution |
| Long-document QA | distant evidence in one document | hierarchy, parent-child, late chunking, or long-context route | boundary and position bias |
| Temporal question | fact valid at a requested time | bitemporal filters + version-aware ranking | latest document answers historical query |
| Conversational follow-up | current turn plus relevant history | rewrite or context-aware retrieval + memory policy | topic drift or stale preference |
| Recommendation | evidence plus user constraints | filtered retrieval + diversity + policy | popularity/retrieval bias |
| Procedural answer | ordered steps with prerequisites | section/graph retrieval + coverage checking | steps assembled in wrong order |
| Numerical/table QA | schemas, rows, cells, operations | table retrieval + executable SQL/calculation | text generator performs unreliable arithmetic |
| Visual QA | page/region/image evidence | visual or multimodal retrieval + VLM | parser/caption erased visual evidence |
| Code task | symbols, dependencies, tests | lexical+dense+graph repo retrieval | obsolete revision or incompatible symbol |
| Unanswerable/false premise | no sufficient evidence | sufficiency estimator + abstention | nearest distractor treated as proof |
| High-stakes advisory | authoritative, current, scoped evidence | allowlisted retrieval + claim audit + human gate | source authority confused with semantic relevance |

The same user request can contain several task types. A production planner may
therefore emit a small retrieval program rather than one query.

## 5. Retrieval-unit design space

Granularity determines both the retriever’s learning problem and the generator’s
evidence burden.

- **Whole document:** retains global coherence but produces coarse matches and
  expensive prompts.
- **Fixed token window:** simple and batchable; can cut propositions, tables,
  lists, and section dependencies.
- **Sentence or paragraph:** natural boundaries; may lack required surrounding
  definitions.
- **Semantic segment:** places a boundary at topic or discourse change; quality
  depends on the segmenter and may drift by domain/language.
- **Proposition:** high retrieval precision and clean claim mapping; extraction
  adds cost and can omit qualifiers or provenance.
- **Parent-child:** retrieve small units but expand to a parent section for
  generation; improves specificity while preserving context at added token cost.
- **Sliding window with sentence center:** embeds a contextual window but returns
  the centered sentence, separating retrieval context from generation context.
- **Late chunking:** contextualize a long document before pooling chunk spans;
  preserves cross-chunk context but is limited by embedding-model context and
  may dilute very long documents.
- **Hierarchical summary:** leaf passages plus recursive abstractions; supports
  global questions but summaries can be lossy and are expensive to update.
- **Graph node or path:** entities, events, propositions, passages, or communities;
  relies on extraction and linking quality.
- **Table unit:** table, row, column, cell group, schema, or a generated textual
  view; must preserve header and unit associations.
- **Visual unit:** page, crop, patch vector, region, figure, or frame; index size
  grows with vector count.
- **Learned memory vector:** compact and fast, but less human-auditable and harder
  to delete or cite.

There is no context-free optimal chunk size. The correct experiment varies
unit construction, retrieval \(k\), expansion policy, and context budget
together because their effects interact.

## 6. Retrieval model families

### 6.1 Lexical and probabilistic

TF-IDF, query likelihood, BM25 and its variants use observable terms and an
inverted index. Pseudo-relevance feedback and RM3 estimate useful expansion
terms from an initial result set. Their strengths are exact identifiers,
interpretability, cheap updates, and metadata/filter integration; their central
weakness is vocabulary mismatch.

### 6.2 Neural term weighting and learned sparse retrieval

DeepCT, docT5query, DeepImpact, uniCOIL, SPLADE, and successors learn term
weights or expansions while retaining sparse postings. They bridge semantic
matching and inverted-index serving but can produce large posting lists, require
regularization, and inherit tokenizer vocabulary limits.

### 6.3 Single-vector dense retrieval

Dual encoders map query and unit independently and rank by dot product or
cosine. Training uses positives and sampled negatives, often with in-batch and
hard-negative mining. DPR, ANCE, RocketQA, Contriever, RetroMAE, E5,
INSTRUCTOR, GTR, BGE, and reasoning-oriented retrievers differ in supervision,
pretraining, instructions, pooling, and negative construction. Dense retrieval
improves paraphrase matching but can miss rare strings and makes index migration
expensive.

### 6.4 Multi-vector and late interaction

ColBERT retains token vectors and scores a query with a MaxSim aggregation.
Visual late-interaction systems such as ColPali apply the same idea to image
patches. Multi-vector methods preserve fine-grained evidence better than one
pooled vector but multiply storage and search cost; compression and candidate
pruning are core parts of the system.

### 6.5 Cross-encoder ranking

A cross-encoder jointly attends over query and candidate, increasing interaction
quality while preventing independent document indexing. It is usually applied
to tens or hundreds of first-stage candidates. Pointwise, pairwise, listwise,
and setwise objectives optimize different ranking properties; a pointwise
relevance score does not ensure complementary evidence coverage.

### 6.6 Hybrid, routed, and federated retrieval

Score interpolation requires calibrated score scales. Reciprocal-rank fusion
uses ranks, while learned fusion may use query features and per-retriever
confidence. Routers choose retrievers by intent, domain, modality, or predicted
utility. Federated retrieval additionally accounts for source cost, latency,
authorization, and result availability.

### 6.7 Structured retrieval

SQL, graph traversal, symbolic filters, APIs, and program execution can answer
questions similarity search cannot. LLMs may plan these operations, but schema
linking, execution errors, permissions, and returned-result validation remain
separate problems.

## 7. Index families and their real trade-offs

Exact search computes every similarity and provides an oracle for measuring ANN
loss. Inverted indexes provide exact term lookup. Dense ANN indexes trade recall
for latency and memory:

- **LSH:** hashes nearby points together with probabilistic guarantees; many
  tables/probes can be required in hard high-dimensional spaces.
- **Tree/partition methods:** recursively restrict candidates; performance
  degrades in high intrinsic dimension.
- **IVF:** searches selected coarse clusters; `nprobe` controls recall/latency.
- **Product quantization:** encodes subvector codebook assignments; saves memory
  at the cost of distance distortion.
- **HNSW:** searches a multilayer proximity graph; strong in-memory performance,
  but build cost and memory grow with graph degree and recall settings.
- **DiskANN/Vamana:** graph traversal designed around SSD access and compressed
  in-memory routing state.
- **SPANN:** keeps centroids in memory and posting lists on disk with closure
  augmentation and query-aware pruning.
- **Multi-vector indexes:** add centroiding, residual compression, token pruning,
  or two-stage search to make MaxSim feasible.

ANN recall must be measured against exact nearest neighbors and then against
task qrels. A high vector recall can still retrieve semantically similar but
non-supporting evidence; an apparently lower ANN recall may have no answer-level
effect if the omitted neighbors were redundant.

## 8. Query-policy families

1. **No transformation:** retain user wording; cheapest and easiest to trace.
2. **Normalization/correction:** spelling, identifiers, dates, and filters.
3. **Classical expansion:** thesauri, Rocchio, pseudo-relevance feedback, RM3.
4. **Generated expansion:** docT5query on documents; Query2Doc or RAG-Fusion
   style variants on queries.
5. **Hypothetical evidence:** HyDE embeds a generated pseudo-document; useful
   zero-shot but can anchor retrieval on generated errors.
6. **Conversation rewrite:** resolve pronouns and ellipsis into a standalone
   query; rewrite errors may silently change intent.
7. **Step-back/generalization:** search a broader principle before the specific
   question.
8. **Decomposition:** create subquestions for entities, hops, comparisons, or
   operations; results need deduplication and coverage-aware fusion.
9. **Iterative feedback:** use retrieved evidence or a partial answer to choose
   the next query, as in IRCoT, FLARE, and ITER-RETGEN.
10. **Adaptive routing:** choose no retrieval, one-shot retrieval, multi-hop,
    graph, long context, web, or tools based on predicted benefit.
11. **Learned search policy:** supervised trajectories or RL learn query,
    retrieve, inspect, and stop actions; rewards must prevent fabricated search
    traces and unnecessary calls.

Every transformation must retain the original request, record generated
queries, limit fan-out, and expose a stop budget. Query improvement is evaluated
by final evidence and answer utility, not linguistic plausibility.

## 9. Evidence selection and context construction

Candidate relevance is necessary but not sufficient. The selected set should
optimize

\[
Z^*=\arg\max_{Z\subseteq R}
\big[\alpha\,\mathrm{support}(Z,x)
+\beta\,\mathrm{coverage}(Z,x)
+\gamma\,\mathrm{authority}(Z)
+\delta\,\mathrm{freshness}(Z,x)
-\lambda\,\mathrm{redundancy}(Z)
-\mu\,\mathrm{risk}(Z)\big]
\]

subject to token, latency, source, and permission constraints. Techniques
include cross-encoder reranking, listwise ranking, maximal marginal relevance,
submodular/set-cover selection, clustering, duplicate collapse, contradiction
grouping, temporal filters, source diversity, parent expansion, and learned
compression.

Context ordering matters because language models exhibit position bias. Common
policies place the strongest evidence first, distribute evidence at both ends,
group by subquestion, preserve document order for procedures, or serialize a
graph/table structure. One ordering is not optimal across models and tasks.

Compression can select sentences, delete low-utility tokens, generate evidence
summaries, or map text into learned vectors. Extractive compression preserves
source spans more easily; abstractive compression can synthesize but creates a
new hallucination and provenance layer. A compression benchmark must report
answer utility, token savings, latency, and claim/citation preservation.

## 10. Generator integration families

- **Extractive reader:** chooses spans; strong provenance but cannot naturally
  synthesize or rephrase.
- **Prompt RAG:** concatenates evidence for a decoder-only or seq2seq model;
  modular but sensitive to prompt, order, and context length.
- **RAG-Sequence/RAG-Token:** marginalizes latent documents for the sequence or
  each token; differentiable over a truncated result set.
- **Fusion-in-Decoder:** independently encodes many passages and fuses them in
  decoder attention; scales evidence count but decoder attention remains costly.
- **Reader/retriever distillation:** transfers evidence utility from a reader to
  retriever scores.
- **Retrieval-augmented pretraining:** REALM, RETRO, Atlas and relatives expose
  the model to retrieval during learning rather than only at inference.
- **REPLUG-style black-box augmentation:** trains a retriever while keeping a
  language model frozen or inaccessible.
- **Interleaved retrieval/decoding:** retrieves when uncertainty or a control
  token triggers, then continues generation.
- **Structured generation:** generates programs, graph paths, SQL, or claims
  before natural-language realization.
- **Multimodal generation:** a VLM consumes pages, regions, images, audio, or
  mixed evidence; citation granularity must match the modality.

Integration quality must be tested with oracle evidence, distractors,
contradictions, shuffled order, and missing evidence. Otherwise retriever and
generator errors remain confounded.

## 11. Learning and optimization taxonomy

### Retriever objectives

- pairwise or listwise supervised ranking;
- multiple-negative softmax / InfoNCE;
- triplet or margin loss;
- in-batch, lexical, mined, adversarial, or cross-encoder negatives;
- unsupervised inverse cloze and contrastive pretraining;
- weak answer-string supervision;
- synthetic query generation and teacher labels;
- reader attention or likelihood distillation;
- instruction-conditioned and multilingual representation learning;
- utility-aware objectives based on downstream answer performance.

False negatives deserve explicit handling. A passage not labeled relevant may
still support the answer; aggressive hard-negative mining can teach the model to
reject valid evidence. Use multiple positives, qrel augmentation, teacher
filtering, false-negative masks, and human inspection.

### Generator and joint objectives

- token likelihood with gold or retrieved evidence;
- latent evidence marginal likelihood;
- retrieval-aware pretraining;
- citation/attribution likelihood;
- contrastive preference between grounded and unsupported answers;
- instruction tuning on retrieve/reason/cite trajectories;
- direct preference optimization for answer or search behavior;
- outcome RL, process rewards, search-cost penalties, and curriculum learning.

Joint training creates credit-assignment risk: a correct answer can be produced
from parametric knowledge while retrieval is wrong, or a weak answer can punish
valid evidence. Intermediate reasoning text is not automatically faithful.
Training should reward support selection, answer correctness, citation
entailment, calibrated abstention, and budget compliance separately.

## 12. Adaptive, corrective, and agentic RAG

The control-policy questions are:

1. Is external knowledge needed?
2. Which source or retriever should be used?
3. What query or structured operation should run?
4. Is the returned evidence relevant, sufficient, trustworthy, and current?
5. Should the system reformulate, broaden, narrow, switch modality, or use long
   context?
6. When is the evidence set complete enough to answer?
7. When must the system abstain or escalate?

Corrective RAG evaluates retrieval and may search elsewhere. Self-RAG emits
retrieval and critique tokens. Adaptive-RAG routes by estimated question
complexity. Search-R1/ReSearch/StepSearch use RL for interleaved search. GRIP
represents retrieval actions inside generation. Q-RAG learns evidence selection
while freezing the LLM. These systems differ in action space, supervision,
retriever, maximum calls, and reward—not merely in the label “agentic.”

An agentic system requires hard limits outside the learned policy: maximum
steps, latency/cost budget, source allowlist, tool permissions, loop detection,
and full action/evidence logs.

## 13. Graph, hierarchical, and structured RAG

Graph RAG has several non-equivalent forms:

- corpus-wide LLM-extracted entity graphs and community summaries;
- passage/entity graphs searched with PageRank;
- curated KGs with symbolic paths;
- document citation/link graphs;
- table/schema graphs;
- query-specific incremental graphs;
- graphs used only for candidate expansion;
- graphs used to organize evidence after text retrieval.

It is useful when relations, hierarchy, multi-hop paths, aggregation, or global
themes are central. It is costly when extraction is noisy, updates are frequent,
or questions are local fact lookups. Evaluation must separate extraction,
entity linking, graph retrieval, text retrieval, and answer generation.

Hierarchical RAG similarly includes parent-child expansion, recursive summary
trees, section indexes, topic clusters, and map-reduce global search. Summaries
can omit facts or introduce unsupported claims, and changing a leaf can require
recomputing ancestors.

## 14. Long context versus retrieval

Long context changes but does not remove the information-selection problem.
When the complete source fits, long context can beat imperfect retrieval because
it avoids a retrieval recall ceiling. Retrieval can win on cost, latency, very
large corpora, access control, and focused evidence density. Hybrid routers first
try retrieval and escalate to long context when evidence is insufficient, or
select a subset while retaining document-wide contextual representations.

Always measure:

- source tokens versus prompt tokens after selection;
- effective evidence position and distractor density;
- retrieval latency plus model time-to-first-token and end-to-end time;
- answer quality at equal cost or latency;
- cache reuse;
- behavior under paraphrase, because keyword retrieval can collapse while long
  context remains stable.

## 15. Grounding, attribution, and abstention

The output must be decomposed into atomic externally verifiable claims. For each
claim, record supporting source ID, immutable version, exact span or region,
retrieval timestamp, and entailment judgment. Evaluate:

- **citation correctness/precision:** cited evidence supports the attached claim;
- **citation completeness/recall:** all claims needing evidence are supported;
- **source quality/authority:** the source is appropriate, not merely entailing;
- **attribution localization:** citation is attached to the right claim;
- **faithfulness:** answer claims follow from provided evidence;
- **factual correctness:** claims are true under the task’s reference and time;
- **abstention calibration:** confidence/coverage predicts when answering is safe.

A citation-looking URL is not evidence. A passage can mention the answer without
supporting it; several passages can conflict; a true claim can be unsupported by
the retrieved context. These are separate labels.

## 16. Evaluation surface

A complete evaluation contains four controlled layers:

1. **Corpus/unit:** parsing accuracy, unit coverage, deduplication, metadata,
   permissions, freshness, and index completeness.
2. **Retrieval/selection:** Recall@k, precision, MRR, nDCG, claim recall, set
   coverage, diversity, ANN recall, and filter correctness.
3. **Oracle-context generator:** correctness, completeness, faithfulness,
   citation entailment, robustness to noise/conflict/order, and abstention.
4. **End-to-end/product:** task success, human preference, latency, cost,
   availability, risk, and failure attribution.

Results must be sliced by answerability, popularity, time sensitivity, query
type, hop count, language, document length, modality, tenant, source authority,
and turn position. Report paired uncertainty, not only mean scores. Freeze
corpus/query time, qrels, code, model and prompt versions, random seeds, and raw
per-query traces.

The benchmark chapter audits KILT, BEIR, MTEB/MMTEB, RAGAS, ARES, RGB,
CRUD-RAG, RAGTruth, RAGChecker, CRAG, BRIGHT, NoMIRACL, mtRAG, GaRAGe,
LongMemEval, and TREC RAG. None measures every layer.

## 17. Security, privacy, and governance surface

Retrieved content is untrusted data. The threat model includes:

- corpus poisoning and targeted retrieval manipulation;
- trigger/backdoor attacks against memories and embeddings;
- indirect prompt injection and tool hijacking;
- malicious advertisements, conflict, and denial-of-service content;
- cross-tenant retrieval and authorization bypass;
- document-existence and membership inference;
- raw corpus, structured graph, image, audio, and prompt leakage;
- embedding inversion and query leakage to remote services;
- cache timing side channels;
- PII, secrets, regulated records, copyright, and license violations;
- stale deletion replicas, derived summaries, graph edges, caches, and logs;
- source spoofing, citation laundering, and low-authority evidence.

Controls include pre-retrieval ACL enforcement, post-retrieval policy checks,
source signatures and hashes, trust-domain separation, active-content
sanitization, tool isolation, canary documents, anomaly monitoring, conflict and
duplicate detection, least privilege, output DLP, immutable audit logs, red-team
suites, and verified deletion across all derived artifacts. No perplexity
filter, paraphrase filter, or single safety model is a complete defense.

## 18. Production and economic surface

The operating objective is constrained utility, not benchmark accuracy:

\[
U = Q
-\lambda_L L_{p95}
-\lambda_C C
-\lambda_R R
-\lambda_F(1-A),
\]

where \(Q\) is task quality, \(L_{p95}\) tail latency, \(C\) monetary/compute
cost, \(R\) risk, and \(A\) availability. Report at least:

- ingestion lag, parse failure, index build/update throughput;
- index bytes per unit, vector count, replication and storage tiers;
- candidate, rerank, prompt, and completion counts;
- p50/p95/p99 retrieval, rerank, time-to-first-token, and end-to-end latency;
- cache hit rates and invalidation age;
- retrieval/tool-call rate and early-stop rate;
- cost per query and per successful, correctly cited answer;
- abstention, fallback, timeout, and partial-failure rates;
- quality/risk metrics by deployment slice and version.

Blue/green indexes, dual reads, shadow evaluation, replayable traces, gradual
rollout, and rollback are required for retriever or embedding migrations. A new
embedding space is normally incompatible with old document vectors; migration
is a data change, not only a model configuration change.

## 19. Domain-specific adaptations

### Biomedical and clinical

Prioritize authoritative source tiers, publication date, evidence grade,
terminology/abbreviations, patient privacy, and mandatory abstention/escalation.
Retrieval correctness does not make generated clinical advice safe.

### Legal and compliance

Preserve jurisdiction, court, authority level, effective date, amendments,
citations, and quoted language. Temporal and authority filters often matter more
than semantic similarity.

### Finance

Distinguish event time, filing time, restatements, market data timestamps, and
derived calculations. Use executable arithmetic and preserve unit/currency.

### Science

Index title/abstract/body, figures, formulas, tables, references, methods, and
supplements. Citation count is not source quality; retractions and versions need
explicit metadata.

### Enterprise support

Combine product/version/platform metadata, known issues, tickets, and runbooks.
Prevent one customer’s ticket or secret from crossing tenants.

### Software engineering

Retrieve against the exact repository revision and language environment. Use
symbols, dependency graphs, issues, tests, and execution feedback; validate code
instead of treating textual similarity as correctness.

### Multilingual and cross-lingual

Choose between multilingual indexes, per-language indexes, query translation,
and cross-lingual retrieval. Evaluate language-specific tokenization, script,
morphology, source availability, answer language, and unequal benchmark depth.

## 20. The minimum complete experiment matrix

At minimum, compare:

1. no retrieval and oracle context;
2. BM25;
3. dense retrieval;
4. sparse+dense hybrid;
5. hybrid plus reranker;
6. two unit sizes and a parent-child or contextual strategy;
7. two evidence budgets and at least one ordering policy;
8. fixed one-shot versus adaptive/iterative retrieval when the task is multi-hop;
9. context-only versus permissive generation and calibrated abstention;
10. clean, irrelevant-noise, conflicting, stale, poisoned, and unauthorized
    evidence conditions;
11. exact dense search versus the intended ANN index on a representative slice;
12. latency/cost at equal quality and quality at equal budget.

Every ablation changes one declared factor, keeps a per-query trace, and reports
paired differences with uncertainty. A component is adopted only if it improves
the target Pareto frontier or satisfies a hard requirement.

## 21. Repository reading map

| Need | Detailed material |
|---|---|
| Historical causality and dates | [`chronology.md`](chronology.md) |
| Every registered work in first-public order | [`chronological_index.md`](chronological_index.md) |
| 2024–2026 peer-reviewed frontier | [`frontier_2024_2026.md`](frontier_2024_2026.md) |
| Corpus, parsing, chunking, metadata, indexes | [`corpus_and_indexing.md`](corpus_and_indexing.md) |
| Sparse, dense, multi-vector, fusion, reranking | [`retrieval_and_ranking.md`](retrieval_and_ranking.md) |
| Query transformation, context, generation, citations | [`context_and_generation.md`](context_and_generation.md) |
| Retriever/generator/joint/RL learning | [`training_and_optimization.md`](training_and_optimization.md) |
| Graph, hierarchy, tables, multimodal, code, domains | [`structured_and_multimodal_rag.md`](structured_and_multimodal_rag.md) |
| Adaptive search, agents, long-term memory, time | [`agents_memory_and_temporal.md`](agents_memory_and_temporal.md) |
| Metrics, benchmarks, statistics, failure tests | [`evaluation_and_risks.md`](evaluation_and_risks.md) |
| Threats, privacy, access control, governance | [`security_privacy_and_governance.md`](security_privacy_and_governance.md) |
| Serving, observability, cost, migration, incidents | [`production_systems.md`](production_systems.md) |
| Equations | [`mathematical_primer.md`](mathematical_primer.md) |
| Task-to-architecture choices | [`decision_guide.md`](decision_guide.md) |
| Terminology | [`glossary.md`](glossary.md) |
| Primary-source registry | [`sources.json`](sources.json) |
| Subject → chapter → notebook → code → test traceability | [`coverage_matrix.md`](coverage_matrix.md) |
| Entire handbook in one Jupyter artifact | [`00_complete_rag_handbook.ipynb`](../notebooks/00_complete_rag_handbook.ipynb) |

The executable notebooks follow the same sequence. Each implementation is a
small transparent model of a mechanism, not an unreported reproduction of a
frontier neural system.