# Evaluating RAG: metrics, benchmarks, failure modes, and risk controls

RAG evaluation is not one score. It is a causal diagnosis over at least three
stages:

```text
query → retrieved candidates → selected evidence → generated claims/citations
          retrieval layer       reader layer          end-to-end layer
```

An end-to-end miss may come from corpus absence, permissions, parsing,
chunking, retrieval, ANN approximation, fusion, reranking, packing, generation,
citation, or abstention. Aggregate answer accuracy cannot locate the failure.

## 1. Metric definitions and what they do not prove

### Retrieval

For relevance set \(G_q\), ranked list \(R_q\), and cutoff \(k\):

\[
P@k=\frac{|R_{q,1:k}\cap G_q|}{k},\qquad
R@k=\frac{|R_{q,1:k}\cap G_q|}{|G_q|}.
\]

Mean reciprocal rank rewards the first relevant item:

\[
\operatorname{MRR}=\frac1{|Q|}\sum_q\frac1{\operatorname{rank}_q^{(1)}}.
\]

With graded gain \(g_i\),

\[
\operatorname{DCG}@k=\sum_{i=1}^k\frac{2^{g_i}-1}{\log_2(i+1)},
\qquad
\operatorname{nDCG}@k=\frac{\operatorname{DCG}@k}{\operatorname{IDCG}@k}.
\]

These metrics depend on qrels. “Answer-containing recall” from early open QA
only checks whether an answer string appears. It can count a contradictory or
irrelevant passage as positive and miss paraphrased support. Passage and page
labels are different; pooled qrels favor systems that contributed to the pool.

For RAG, add:

- **claim recall:** fraction of reference claims whose supporting evidence was
  retrieved;
- **context precision:** fraction of supplied evidence supporting a needed
  claim;
- **evidence utility:** answer quality difference when a candidate is included
  versus withheld;
- **coverage under budget:** recall at a fixed token, latency, and dollar budget;
- **source/permission/freshness validity:** whether retrieved evidence was
  authorized and temporally valid.

### Generation

Exact match and token F1 are useful for short answers but punish valid
paraphrases and ignore explanation quality. ROUGE/BLEU measure surface overlap,
not factuality. For long-form RAG, decompose the answer into atomic claims
\(A=\{a_i\}\) and references \(Y=\{y_j\}\):

- claim precision: supported/correct generated claims divided by generated
  claims;
- claim recall/completeness: covered reference claims divided by reference
  claims;
- answer relevance/directness;
- contradiction and unsupported-introduction rates;
- answerability, abstention, and selective risk at coverage \(c\).

If a system answers only easy queries, accuracy can rise while usefulness
falls. Always plot risk/accuracy against answer coverage and report false-answer
versus unnecessary-abstention rates separately.

### Citations and attribution

For each externally verifiable claim, distinguish:

1. **citation completeness/recall:** did the claim receive a citation?
2. **citation entailment/precision:** does the cited span support that claim?
3. **source quality/authority:** should this source be trusted for the claim?
4. **provenance validity:** is the ID immutable, accessible, licensed, and the
   same version the model saw?
5. **causal use:** did the evidence influence the answer, rather than merely
   being appended afterward?

No automatic metric proves all five. A hyperlink next to a true statement can
still point to an irrelevant page. A perfectly entailing malicious page is not
authoritative. Retrieved evidence accompanying an answer is not necessarily
causal attribution.

### Cost and systems metrics

Report at minimum:

- p50/p95/p99 retrieval, rerank, time-to-first-token, and end-to-end latency;
- sparse/dense/graph/image index bytes per document and total resident memory;
- ingest, update, deletion, and rebuild time;
- candidates retrieved/reranked, evidence tokens, model input/output tokens,
  tool calls, retries, and cache hit rate;
- CPU/GPU hours and dollars per query;
- **cost per correct supported answer**, not cost per attempt;
- tail failure rates under concurrent load.

---

## 2. Benchmark and evaluator audit

These resources answer different questions. Do not compare them as one
leaderboard.

### KILT: shared knowledge snapshot with provenance gating

**Status:** NAACL 2021 peer-reviewed.

KILT unifies 11 datasets—FEVER, Natural Questions, HotpotQA, TriviaQA, ELI5,
Wizard of Wikipedia, T-REx, AIDA and others—on the 2019-08-01 Wikipedia
snapshot. Page R-precision is \(r/R\); Recall@\(k\) handles alternative
provenance sets; a KILT task score is zero unless provenance R-precision equals
one.

Original RAG KILT scores included FEVER 53.45, NQ 32.69, Hotpot 3.21, TriviaQA
38.13, ELI5 1.69, and WoW 9.1. The benchmark is historically important but
static, English Wikipedia-only, page-level, incomplete in provenance, and
all-or-nothing. Report retrieval, downstream, and gated scores together.
[Paper](https://aclanthology.org/2021.naacl-main.200/).

### BEIR: heterogeneous zero-shot retrieval

**Status:** NeurIPS 2021 Datasets & Benchmarks.

BEIR originally contains 18 retrieval datasets across domains and task types,
with nDCG@10 primary and MAP/MRR/precision/recall secondary. BM25 followed by a
MiniLM reranker beat BM25 on 16/18 and averaged roughly 11% improvement; many
dense models lost to BM25. Additional TREC-COVID judgments materially changed
ANCE's ranking, demonstrating pooled-qrel bias.

BEIR is retrieval-only, largely English, short-query, and static. Use it to
measure out-of-domain robustness and retain BM25/hybrid controls; do not infer
grounded generation. [Paper](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/65b9eea6e1cc6bb9f0cd2a47751a186f-Abstract-round2.html).

### MTEB and MMTEB: representation quality is multi-task and multilingual

**Status:** MTEB EACL 2023; MMTEB ICLR 2025.

MTEB's original release covers 58 datasets, eight task categories, 112
languages, and 33 models. Retrieval uses nDCG@10; reranking uses MAP/MRR. The
paper's global averages had ST5-XXL 59.51, GTR-XXL 58.97,
SGPT-5.8B-msmarco 58.81, and all-mpnet 57.78; SGPT was strongest in retrieval
while STS specialists transferred poorly. A global embedding average is not a
domain retriever choice.

MMTEB expands to over 500 quality-controlled tasks, ten categories, 250+
languages, long documents, code, reasoning, and instruction following, using
per-task metrics and Borda aggregation. Dataset counts differ among paper
sections/releases, so pin the exact suite. [MTEB paper](https://aclanthology.org/2023.eacl-main.148/) and
[MMTEB paper](https://proceedings.iclr.cc/paper_files/paper/2025/file/fc0e3f908a2116ba529ad0a1530a3675-Paper-Conference.pdf).

### RAGAS: cheap reference-free diagnostics

**Status:** EACL 2024 demo paper; package API has evolved.

The original metrics are:

- faithfulness \(F=\) context-supported answer claims / answer claims;
- answer relevance: mean embedding similarity between the actual question and
  questions generated from the answer;
- context relevance: extracted relevant context sentences / all context
  sentences.

WikiEval contains only 50 synthetic current-event Wikipedia questions. Human
pairwise agreement was 0.95 for faithfulness, 0.78 answer relevance, and 0.70
context relevance, versus GPT-score 0.72/0.52/0.63. RAGAS is useful for a pinned
smoke test and regression trend. Answer relevance is not correctness; judges,
prompts, embeddings, temperatures, and package versions must be recorded.
[EACL paper](https://aclanthology.org/2024.eacl-demo.16/).

### ARES: calibrated aggregate evaluator rates

**Status:** NAACL 2024 long paper.

ARES uses FLAN-T5-XXL to synthesize positive/negative examples and
DeBERTa-v3-Large judges for context relevance, faithfulness, and answer
relevance. It needs at least five in-domain demonstrations and roughly 150
labeled examples, often 300. Prediction-powered inference estimates an
aggregate rate:

\[
\hat\mu=
\frac1N\sum_{i=1}^{N}f(x_i)
+\frac1n\sum_{j=1}^{n}[y_j-f(x_j)],
\]

with confidence intervals from the labeled correction term.

Across eight tasks from KILT/SuperGLUE/AIS, reported evaluator accuracy improved
59.3 points over RAGAS context relevance and 14.4 for answer relevance;
aggregate hallucination estimates were within 2.5 points while using 78% fewer
labels; system-ranking correlations were generally 0.82–1.0. ARES is strong for
stable-domain release comparison. PPI calibrates aggregate rates, not each
answer; expert labels and GPU training remain necessary.
[NAACL paper](https://aclanthology.org/2024.naacl-long.20/).

### RGB: controlled noise, rejection, integration, and counterfactuals

**Status:** AAAI 2024 peer-reviewed.

RGB has 600 recent-news base questions, 200 information-integration questions,
and 200 counterfactual cases in English/Chinese with five controlled documents.
It measures noise robustness, negative rejection, information integration, and
counterfactual robustness with containment accuracy, rejection, error
detection, and correction.

ChatGPT accuracy fell 96.33% → 76% as noise rose 0 → 80%; best rejection was
only 45% English/43.33% Chinese. Counterfactual evidence drove English ChatGPT
from 89% closed/no-document accuracy to 9%. RGB primarily stress-tests a
generator given controlled context, not a real retriever; false documents are
generated and string metrics are brittle. Recreate these perturbations on the
product corpus. [AAAI paper](https://ojs.aaai.org/index.php/AAAI/article/view/29728).

### CRUD-RAG: configurations depend on operation type

**Status:** 2024 preprint; later ACM TOIS article.

The Chinese-news benchmark contains Create/continuation 10,728; Read 3,199
one-document, 3,192 two-document, 3,189 three-document; Update/correction 5,130;
Delete/multi-document summary 10,728—36,166 cases over an 86,834-item corpus.
Metrics include BLEU, ROUGE-L, BERTScore, MRR, and RAGQuestEval.

No configuration wins universally: hybrid+rerank is usually strongest; Create
favors larger overlapping chunks; multi-document Read benefits from larger
\(k\); BM25 remains competitive for summary/update. Expensive experiments used
only one fifth, cases are generated, and overlap/LLM metrics are limited. Report
each CRUD operation separately. [Paper](https://arxiv.org/abs/2401.17043).

### RAGTruth: hallucination spans under supplied context

**Status:** ACL 2024 long paper.

RAGTruth has 2,965 prompts and 17,790 responses from six 2023 models across QA,
data-to-text, and summarization; 14,289 hallucinated spans; 7,664 responses
(43.1%) contain at least one. Labels separate evident/subtle conflict and
unsupported introduction.

A fine-tuned Llama-2-13B detector reported response F1 78.7 versus prompted
GPT-4 63.4; span F1 was only 52.7 versus 28.3. Data-to-text response
hallucination reached 68.6%. This evaluates detection/grounding, not retrieval;
a strict context-only policy labels a true external fact unsupported.
[ACL paper](https://aclanthology.org/2024.acl-long.585/).

### RAGChecker: claim-level component diagnosis

**Status:** NeurIPS 2024 Datasets & Benchmarks.

RAGChecker covers 4,162 questions from ten English domains. It decomposes
answers into claims and reports claim precision/recall/F1, retriever claim
recall/context precision, generator faithfulness, context utilization, relevant
and irrelevant noise sensitivity, hallucination, and correct unsupported
self-knowledge.

Human-correlation Pearson/Spearman was 0.6193/0.6090 versus strongest reported
RAGAS answer-similarity 0.4831/0.5723. Increasing \(k\) 5 → 20 raised claim
recall 61.5 → 77.6 but also noise sensitivity. It is a strong public diagnostic,
but LLM claim extraction/checking is costly and biased. Human-audit disputed and
high-risk cases. [NeurIPS paper](https://proceedings.neurips.cc/paper_files/paper/2024/file/27245589131d17368cccdfa990cbf16e-Paper-Datasets_and_Benchmarks_Track.pdf).

### CRAG benchmark: dynamic, popular/long-tail, and complex facts

**Status:** NeurIPS 2024 Datasets & Benchmarks; distinct from Corrective RAG.

The Comprehensive RAG Benchmark contains 4,409 English questions across
finance, sports, music, movies, and open domain; eight complexity types including
false premises; about 220k pages, 2.6M knowledge-graph entities, and 38 mock
APIs. Questions span popular to long-tail and facts changing over years to
seconds. Scoring assigns +1 accurate, 0 missing, −1 incorrect; human perfect and
acceptable are +1/+0.5.

GPT-4-Turbo without retrieval reported 33.5% accuracy/13.5% hallucination;
end-to-end RAG 43.6%/30.1%. The benchmark paper notes advanced LMs at or below
34% and industry RAG answering only 63% without hallucination. High dynamism,
low popularity, and complexity are hardest. Preserve time and data snapshots and
report accurate, hallucinated, and missing separately.
[NeurIPS paper](https://proceedings.neurips.cc/paper_files/paper/2024/hash/1435d2d0fca85a84d83ddcb754f58c29-Abstract-Datasets_and_Benchmarks_Track.html).

### BRIGHT: reasoning-intensive retrieval

**Status:** ICLR 2025 peer-reviewed.

The final paper contains 1,384 questions across 12 StackExchange, code, and
theorem domains (earlier versions mention 1,398). Primary metric is nDCG@10.
An embedding model averaging 59.0 on MTEB scored 18.3 on BRIGHT; standard
retrievers peaked around 24.3; explicit query reasoning added up to 12.2 but the
best remained below 30. Use BRIGHT for indirect technical evidence, not
end-to-end answer grounding. [ICLR paper](https://proceedings.iclr.cc/paper_files/paper/2025/file/7a0f8055c838df8e62329a76c7c6403d-Paper-Conference.pdf).

### NoMIRACL: hallucination and miss are separate

**Status:** Findings of EMNLP 2024.

NoMIRACL contains 56,057 contexts across 18 languages, ten language families,
and 11 scripts. Hallucination rate is false-positive answers when no relevant
passage exists; miss rate is failure to answer when evidence exists. Mixtral
had the lowest reported mean hallucination at 17.4%; GPT-4 gave the best
trade-off. Requiring explanations cut hallucination 9.7 points but increased
misses 8.3. Always report both; it tests answer-versus-abstain, not answer
correctness. [EMNLP paper](https://aclanthology.org/2024.findings-emnlp.730/).

### mtRAG and MTRAGEval: multi-turn compounding failures

**Status:** mtRAG TACL 2025; MTRAGEval SemEval 2026.

mtRAG has 110 human conversations, mean 7.7 turns and 842 tasks from Wikipedia,
finance, government, and cloud, labeled answerable/partial/unanswerable/
conversational. Retrieval uses recall/nDCG@1/3/5/10; generation assesses
faithfulness, appropriateness, naturalness, and completeness.

ELSER plus rewriting reported recall@10 0.64 and nDCG@10 0.54. Recall@5 fell
from 0.89 on first turns to 0.47 later; the best automatic metric's Spearman
against human win rate was only about 0.33. SemEval-2026's best retrieval
nDCG@5 was 0.578, gold-context generation composite 0.783, and full-RAG 0.586,
showing compounding errors.
[mtRAG](https://aclanthology.org/2025.tacl-1.36/) and
[MTRAGEval](https://aclanthology.org/2026.semeval-1.447/).

### GaRAGe: human labels expose weak factuality and deflection

**Status:** Findings of ACL 2025.

GaRAGe contains 2,366 questions and more than 35k human-annotated grounding
passages from web and private documents, including dynamic facts and
insufficient evidence. Tested frontier models reached at most 60% relevance-aware
factuality, 31% true-positive deflection, and 58.9% F1 attribution to relevant
sources. Time-sensitive and sparse private evidence was harder. It is strong
evidence that citation-looking answers remain unreliable.
[ACL paper](https://aclanthology.org/2025.findings-acl.875/).

### LongMemEval: retrieval is only one memory operation

**Status:** ICLR 2025 peer-reviewed.

LongMemEval has 500 questions over interactions totaling about 115k to 1.5M
tokens and tests information extraction, cross-session reasoning, updates,
temporal reasoning, and abstention. The key system lesson is that memory write
policy, consolidation, overwrite/update semantics, and forgetting matter—not
just ANN recall. [ICLR paper](https://openreview.net/forum?id=pZiyCaVuti).

### TREC RAG: external adjudication and sentence-level citations

**Status:** annual NIST program, 2024–2026.

The track separates retrieval, generation with organizer-provided context, and
full RAG on MS MARCO V2.1. Pooled relevance judgments, assessor-edited nuggets,
nugget coverage, fluency, citation need, and sentence-level citation support
provide stronger external adjudication than self-judged demos. Pin track year,
corpus, topics, and judgments; pooled evaluation still has incompleteness.
[Official track](https://trec-rag.github.io/) and
[2024 judgments](https://trec.nist.gov/data/rag2024.html).

### Preprint-only evaluators

RAGBench remains a preprint: about 100k examples across 12 datasets with TRACe
relevance/utilization/completeness/adherence, evaluator AUROC roughly
0.64–0.87, but human validation only 40 DelucionQA cases and many GPT-4 labels.
Use only with this status caveat. [Preprint](https://arxiv.org/abs/2407.11005).

---

## 3. A defensible product evaluation protocol

### Build a stratified gold set

Start with 300–1,000 real queries, increasing for high-risk or heterogeneous
products. Preserve natural frequency but oversample failure-critical slices:

- answerable, partially answerable, and unanswerable;
- popular and long-tail entities;
- fresh/dynamic and temporally scoped facts;
- single-hop, multi-hop, comparison, aggregation, and global synthesis;
- conflicts, counterfactuals, false premises, distractors, and duplicated text;
- long documents, tables, diagrams, scans, and layout;
- languages, scripts, and code-switching;
- first-turn, follow-up, correction, and topic-switch conversation;
- access-control roles/tenants and deleted content;
- malicious instructions, poisoned text, advertisements, and denial-of-service
  content.

Each example should have answerability, atomic reference claims, acceptable
answers, evidence spans/source IDs, temporal validity, and authority labels.
Double-label a subset, adjudicate disagreements, and report inter-annotator
agreement.

### Evaluate three layers

1. **Retrieval:** qrels Recall@\(k\), nDCG/MRR, context precision, claim recall,
   permission and temporal validity, ANN versus exact loss, latency/index cost.
2. **Oracle-context generation:** give gold evidence and measure claim P/R/F1,
   completeness, contradiction, citation entailment, abstention, and robustness
   to reordered/noisy/conflicting evidence. This isolates the reader.
3. **End-to-end:** all above plus answer utility, failure attribution,
   freshness, security, p95 latency, token/tool usage, and dollars per supported
   answer.

### Use uncertainty and evaluator audits

- paired bootstrap or approximate randomization for per-query deltas;
- confidence intervals around aggregate rates; ARES/PPI when its assumptions
  and domain labels fit;
- tag/slice results before an overall mean;
- judge swaps and prompt sensitivity;
- 10–20% human audit of high-risk, system-disagreement, and judge-disagreement
  cases;
- multiple seeds for stochastic graph construction, retrieval agents, and
  generated query expansion.

Choose quality gates first, then plot a cost/latency Pareto frontier. A 0.5-point
mean gain cannot compensate for unauthorized retrieval or a large high-stakes
hallucination regression.

---

## 4. Freshness is an operational property

Retrieval makes updating possible; it does not make the index current.

### FreshQA/FreshPrompt

**Status:** Findings of ACL 2024.

FreshQA includes dynamic and false-premise questions with more than 50k human
judgments. FreshPrompt retrieves, curates, and organizes live search results and
beats compared search-prompting/commercial systems in the paper; FreshEval is
an autorater. The benchmark inevitably drifts. Store query time, pages, snippets,
answer, and judgments; a score without its snapshot cannot be replayed.
[ACL paper](https://aclanthology.org/2024.findings-acl.813/).

### Required controls

- change-data-capture and ingest SLA per source;
- event time and valid time, not only ingestion timestamp;
- version IDs/content hashes on every chunk and citation;
- temporal filtering, with decay only where newer is inherently better;
- explicit current-versus-historical query intent;
- conflicting-version detection and authority rules;
- index, summary, embedding, graph, and cache invalidation after corrections;
- deletion/unlearning propagation with verification;
- immutable snapshot replay for evaluation;
- monitoring for retrieval age, stale-answer rate, source lag, and temporal
  leakage.

---

## 5. Security: retrieval adds a hostile data plane

The threat model must include an attacker who can modify, publish, or trigger
retrieval of content—not only an attacker who writes the user prompt.

### AgentPoison

**Status:** NeurIPS 2024 peer-reviewed.

AgentPoison optimizes a trigger so triggered instructions land near poisoned
memory embeddings. With below 0.1% poison rate it reports at least 80% attack
success across driving, QA, and EHR agents with at most 1% benign degradation.
The attack targets memory retrieval and downstream agent actions, showing that
a tiny poisoned region can become a backdoor.
[NeurIPS paper](https://proceedings.neurips.cc/paper_files/paper/2024/hash/eb113910e9c3f6242541c1652e30dfd6-Abstract-Conference.html).

### PoisonedRAG

**Status:** USENIX Security 2025 peer-reviewed.

PoisonedRAG crafts retrieval-oriented and generation-oriented subtexts. Five
malicious texts per target inserted among millions produced about 90% overall
attack success; the Natural Questions example reached 97% in a database with
2,681,468 clean texts. Black- and white-box settings are considered.
Paraphrasing and perplexity filters were insufficient.
[USENIX paper](https://www.usenix.org/conference/usenixsecurity25/presentation/zou-poisonedrag).

### SafeRAG and “RAG LLMs are not safer”

**SafeRAG (ACL 2025)** is a manually constructed Chinese benchmark covering
silver noise, inter-context conflict, soft advertisements, and white
denial-of-service. Fourteen tested RAG components remained vulnerable and
obvious attacks bypassed retrievers, filters, and LMs.
[Paper](https://aclanthology.org/2025.acl-long.230/).

**RAG LLMs Are Not Safer (NAACL 2025)** evaluates 11 LMs and finds retrieval can
reduce safety; even safe model plus nominally safe documents can yield unsafe
output, and ordinary red-team attacks transfer poorly, requiring RAG-specific
tests. [Paper](https://aclanthology.org/2025.naacl-long.281/).

### Defenses and their scope

- **SeCon-RAG (NeurIPS 2025)** filters with entity-intent-relation semantics and
  clustering, then checks answer/evidence conflicts. It is defense in depth, not
  a certificate. [Paper](https://proceedings.neurips.cc/paper_files/paper/2025/hash/668563ef18fbfef0b66af491ea334d5f-Abstract-Conference.html).
- **C-RAG (ICML 2024)** applies conformal risk control, producing an upper
  confidence bound for a bounded generation-risk function under stated
  calibration/distribution-shift assumptions and sufficient conditions where
  RAG risk is below a vanilla LM. It certifies aggregate calibrated risk, not
  corpus provenance or prompt-injection safety.
  [Paper](https://proceedings.mlr.press/v235/kang24a.html).
- **PRA-RAG (Findings ACL 2026)** samples combinations of retrieved texts,
  identifies a robust subset from embedding geometry, aggregates a stable
  representation, and derives poisoned-content impact bounds. It reports attack
  success as low as 1% while retaining 71% accuracy across tested settings.
  Guarantees are threat-model-specific and add sampling/generation cost.
  [Paper](https://aclanthology.org/2026.findings-acl.1794/).
- **Knowledgeable-R1** addresses misleading/counterfactual context but does not
  authenticate sources or block indirect instructions.

### Required system mitigations

1. Treat retrieved bytes as untrusted **data**, never instructions; delimit and
   quote them in the prompt.
2. Preserve trust domain, signature, owner, tenant, ACL, source, version, and
   content hash before indexing.
3. Apply tenant/row access controls before ANN candidate exposure and again
   after reranking; do not rely on prompt instructions for authorization.
4. Sanitize active HTML/scripts/files and isolate evidence processing from
   tool/action privileges.
5. Use source allowlists/quarantine, duplicate and cluster analysis, conflict
   detection, and canary poison documents.
6. Monitor unusual retrieval neighborhoods, trigger-specific concentration,
   citation domains, tool actions, and answer/evidence divergence.
7. Keep immutable audit logs of query, candidates, scores, evidence hashes,
   model/prompt, actions, and response.
8. Test targeted poison, embedding backdoors, indirect prompt injection,
   conflicting sources, ads, DoS text, membership leakage, and exfiltration.
9. Separate the language model's read-only evidence role from any tool with
   network, filesystem, database, or messaging authority.

No perplexity, paraphrase, embedding-distance, or single LLM filter is a
complete defense.

---

## 6. Citation implementation checklist

- Segment the answer into atomic claims before final rendering.
- Require every externally verifiable claim to map to an immutable source ID and
  exact span/page/region.
- Store the source version, retrieval timestamp, valid time, and content hash.
- Evaluate completeness separately from entailment and authority.
- Penalize “citation laundering”: citing a retrieved but non-supporting passage.
- Detect multiple citations that all derive from the same upstream source.
- Surface conflicts and temporal scope rather than silently selecting one.
- Verify links/objects remain accessible under the viewer's permissions.
- Human-review high-stakes health, legal, financial, safety, and operational
  actions.
- Retain the evidence shown to the model for audit even if the live source later
  changes, subject to retention and deletion law.

---

## 7. Minimum reproducibility manifest

```yaml
experiment_id: immutable-id
query_set:
  version: ...
  hash: ...
  slice_schema: ...
corpus:
  sources: [...]
  snapshot_time: ...
  hash: ...
  permissions_version: ...
ingestion:
  parser: ...
  chunker: ...
  chunk_size: ...
  overlap: ...
retrieval:
  sparse_index: ...
  embedding_model_revision: ...
  ann_parameters: ...
  fusion: ...
  reranker_revision: ...
  candidate_k: ...
  evidence_budget_tokens: ...
generation:
  model_revision: ...
  prompt_hash: ...
  decoding: ...
evaluation:
  qrels_version: ...
  metrics: [...]
  judge_model_and_prompt: ...
  human_sample: ...
  confidence_interval: paired-bootstrap
trace_artifacts:
  retrieved_texts_scores_ranks: ...
  answer_claims_and_citations: ...
  latency_tokens_cost: ...
```

Without this manifest, a RAG result can be impossible to distinguish from
corpus drift, a model API update, a changed top-\(k\), or evaluator drift.