Evaluating RAG: metrics, benchmarks, failure modes, and risk controls

RAG evaluation is not one score. It is a causal diagnosis over at least three stages:

query → retrieved candidates → selected evidence → generated claims/citations
          retrieval layer       reader layer          end-to-end layer

An end-to-end miss may come from corpus absence, permissions, parsing, chunking, retrieval, ANN approximation, fusion, reranking, packing, generation, citation, or abstention. Aggregate answer accuracy cannot locate the failure.

1. Metric definitions and what they do not prove

Retrieval

For relevance set \(G_q\), ranked list \(R_q\), and cutoff \(k\):

Mean reciprocal rank rewards the first relevant item:

With graded gain \(g_i\),

These metrics depend on qrels. “Answer-containing recall” from early open QA only checks whether an answer string appears. It can count a contradictory or irrelevant passage as positive and miss paraphrased support. Passage and page labels are different; pooled qrels favor systems that contributed to the pool.

For RAG, add:

• claim recall: fraction of reference claims whose supporting evidence was retrieved;
• context precision: fraction of supplied evidence supporting a needed claim;
• evidence utility: answer quality difference when a candidate is included versus withheld;
• coverage under budget: recall at a fixed token, latency, and dollar budget;
• source/permission/freshness validity: whether retrieved evidence was authorized and temporally valid.

Generation

Exact match and token F1 are useful for short answers but punish valid paraphrases and ignore explanation quality. ROUGE/BLEU measure surface overlap, not factuality. For long-form RAG, decompose the answer into atomic claims \(A=\{a_i\}\) and references \(Y=\{y_j\}\):

• claim precision: supported/correct generated claims divided by generated claims;
• claim recall/completeness: covered reference claims divided by reference claims;
• answer relevance/directness;
• contradiction and unsupported-introduction rates;
• answerability, abstention, and selective risk at coverage \(c\).

If a system answers only easy queries, accuracy can rise while usefulness falls. Always plot risk/accuracy against answer coverage and report false-answer versus unnecessary-abstention rates separately.

Citations and attribution

For each externally verifiable claim, distinguish:

1. citation completeness/recall: did the claim receive a citation?
2. citation entailment/precision: does the cited span support that claim?
3. source quality/authority: should this source be trusted for the claim?
4. provenance validity: is the ID immutable, accessible, licensed, and the same version the model saw?
5. causal use: did the evidence influence the answer, rather than merely being appended afterward?

No automatic metric proves all five. A hyperlink next to a true statement can still point to an irrelevant page. A perfectly entailing malicious page is not authoritative. Retrieved evidence accompanying an answer is not necessarily causal attribution.

Cost and systems metrics

Report at minimum:

• p50/p95/p99 retrieval, rerank, time-to-first-token, and end-to-end latency;
• sparse/dense/graph/image index bytes per document and total resident memory;
• ingest, update, deletion, and rebuild time;
• candidates retrieved/reranked, evidence tokens, model input/output tokens, tool calls, retries, and cache hit rate;
• CPU/GPU hours and dollars per query;
• cost per correct supported answer, not cost per attempt;
• tail failure rates under concurrent load.

2. Benchmark and evaluator audit

These resources answer different questions. Do not compare them as one leaderboard.

KILT: shared knowledge snapshot with provenance gating

Status: NAACL 2021 peer-reviewed.

KILT unifies 11 datasets—FEVER, Natural Questions, HotpotQA, TriviaQA, ELI5, Wizard of Wikipedia, T-REx, AIDA and others—on the 2019-08-01 Wikipedia snapshot. Page R-precision is \(r/R\); Recall@\(k\) handles alternative provenance sets; a KILT task score is zero unless provenance R-precision equals one.

Original RAG KILT scores included FEVER 53.45, NQ 32.69, Hotpot 3.21, TriviaQA 38.13, ELI5 1.69, and WoW 9.1. The benchmark is historically important but static, English Wikipedia-only, page-level, incomplete in provenance, and all-or-nothing. Report retrieval, downstream, and gated scores together. Paper.

BEIR: heterogeneous zero-shot retrieval

Status: NeurIPS 2021 Datasets & Benchmarks.

BEIR originally contains 18 retrieval datasets across domains and task types, with nDCG@10 primary and MAP/MRR/precision/recall secondary. BM25 followed by a MiniLM reranker beat BM25 on 16/18 and averaged roughly 11% improvement; many dense models lost to BM25. Additional TREC-COVID judgments materially changed ANCE's ranking, demonstrating pooled-qrel bias.

BEIR is retrieval-only, largely English, short-query, and static. Use it to measure out-of-domain robustness and retain BM25/hybrid controls; do not infer grounded generation. Paper.

MTEB and MMTEB: representation quality is multi-task and multilingual

Status: MTEB EACL 2023; MMTEB ICLR 2025.

MTEB's original release covers 58 datasets, eight task categories, 112 languages, and 33 models. Retrieval uses nDCG@10; reranking uses MAP/MRR. The paper's global averages had ST5-XXL 59.51, GTR-XXL 58.97, SGPT-5.8B-msmarco 58.81, and all-mpnet 57.78; SGPT was strongest in retrieval while STS specialists transferred poorly. A global embedding average is not a domain retriever choice.

MMTEB expands to over 500 quality-controlled tasks, ten categories, 250+ languages, long documents, code, reasoning, and instruction following, using per-task metrics and Borda aggregation. Dataset counts differ among paper sections/releases, so pin the exact suite. MTEB paper and MMTEB paper.

RAGAS: cheap reference-free diagnostics

Status: EACL 2024 demo paper; package API has evolved.

The original metrics are:

• faithfulness \(F=\) context-supported answer claims / answer claims;
• answer relevance: mean embedding similarity between the actual question and questions generated from the answer;
• context relevance: extracted relevant context sentences / all context sentences.

WikiEval contains only 50 synthetic current-event Wikipedia questions. Human pairwise agreement was 0.95 for faithfulness, 0.78 answer relevance, and 0.70 context relevance, versus GPT-score 0.72/0.52/0.63. RAGAS is useful for a pinned smoke test and regression trend. Answer relevance is not correctness; judges, prompts, embeddings, temperatures, and package versions must be recorded. EACL paper.

ARES: calibrated aggregate evaluator rates

Status: NAACL 2024 long paper.

ARES uses FLAN-T5-XXL to synthesize positive/negative examples and DeBERTa-v3-Large judges for context relevance, faithfulness, and answer relevance. It needs at least five in-domain demonstrations and roughly 150 labeled examples, often 300. Prediction-powered inference estimates an aggregate rate:

with confidence intervals from the labeled correction term.

Across eight tasks from KILT/SuperGLUE/AIS, reported evaluator accuracy improved 59.3 points over RAGAS context relevance and 14.4 for answer relevance; aggregate hallucination estimates were within 2.5 points while using 78% fewer labels; system-ranking correlations were generally 0.82–1.0. ARES is strong for stable-domain release comparison. PPI calibrates aggregate rates, not each answer; expert labels and GPU training remain necessary. NAACL paper.

RGB: controlled noise, rejection, integration, and counterfactuals

Status: AAAI 2024 peer-reviewed.

RGB has 600 recent-news base questions, 200 information-integration questions, and 200 counterfactual cases in English/Chinese with five controlled documents. It measures noise robustness, negative rejection, information integration, and counterfactual robustness with containment accuracy, rejection, error detection, and correction.

ChatGPT accuracy fell 96.33% → 76% as noise rose 0 → 80%; best rejection was only 45% English/43.33% Chinese. Counterfactual evidence drove English ChatGPT from 89% closed/no-document accuracy to 9%. RGB primarily stress-tests a generator given controlled context, not a real retriever; false documents are generated and string metrics are brittle. Recreate these perturbations on the product corpus. AAAI paper.

CRUD-RAG: configurations depend on operation type

Status: 2024 preprint; later ACM TOIS article.

The Chinese-news benchmark contains Create/continuation 10,728; Read 3,199 one-document, 3,192 two-document, 3,189 three-document; Update/correction 5,130; Delete/multi-document summary 10,728—36,166 cases over an 86,834-item corpus. Metrics include BLEU, ROUGE-L, BERTScore, MRR, and RAGQuestEval.

No configuration wins universally: hybrid+rerank is usually strongest; Create favors larger overlapping chunks; multi-document Read benefits from larger \(k\); BM25 remains competitive for summary/update. Expensive experiments used only one fifth, cases are generated, and overlap/LLM metrics are limited. Report each CRUD operation separately. Paper.

RAGTruth: hallucination spans under supplied context

Status: ACL 2024 long paper.

RAGTruth has 2,965 prompts and 17,790 responses from six 2023 models across QA, data-to-text, and summarization; 14,289 hallucinated spans; 7,664 responses (43.1%) contain at least one. Labels separate evident/subtle conflict and unsupported introduction.

A fine-tuned Llama-2-13B detector reported response F1 78.7 versus prompted GPT-4 63.4; span F1 was only 52.7 versus 28.3. Data-to-text response hallucination reached 68.6%. This evaluates detection/grounding, not retrieval; a strict context-only policy labels a true external fact unsupported. ACL paper.

RAGChecker: claim-level component diagnosis

Status: NeurIPS 2024 Datasets & Benchmarks.

RAGChecker covers 4,162 questions from ten English domains. It decomposes answers into claims and reports claim precision/recall/F1, retriever claim recall/context precision, generator faithfulness, context utilization, relevant and irrelevant noise sensitivity, hallucination, and correct unsupported self-knowledge.

Human-correlation Pearson/Spearman was 0.6193/0.6090 versus strongest reported RAGAS answer-similarity 0.4831/0.5723. Increasing \(k\) 5 → 20 raised claim recall 61.5 → 77.6 but also noise sensitivity. It is a strong public diagnostic, but LLM claim extraction/checking is costly and biased. Human-audit disputed and high-risk cases. NeurIPS paper.

CRAG benchmark: dynamic, popular/long-tail, and complex facts

Status: NeurIPS 2024 Datasets & Benchmarks; distinct from Corrective RAG.

The Comprehensive RAG Benchmark contains 4,409 English questions across finance, sports, music, movies, and open domain; eight complexity types including false premises; about 220k pages, 2.6M knowledge-graph entities, and 38 mock APIs. Questions span popular to long-tail and facts changing over years to seconds. Scoring assigns +1 accurate, 0 missing, −1 incorrect; human perfect and acceptable are +1/+0.5.

GPT-4-Turbo without retrieval reported 33.5% accuracy/13.5% hallucination; end-to-end RAG 43.6%/30.1%. The benchmark paper notes advanced LMs at or below 34% and industry RAG answering only 63% without hallucination. High dynamism, low popularity, and complexity are hardest. Preserve time and data snapshots and report accurate, hallucinated, and missing separately. NeurIPS paper.

BRIGHT: reasoning-intensive retrieval

Status: ICLR 2025 peer-reviewed.

The final paper contains 1,384 questions across 12 StackExchange, code, and theorem domains (earlier versions mention 1,398). Primary metric is nDCG@10. An embedding model averaging 59.0 on MTEB scored 18.3 on BRIGHT; standard retrievers peaked around 24.3; explicit query reasoning added up to 12.2 but the best remained below 30. Use BRIGHT for indirect technical evidence, not end-to-end answer grounding. ICLR paper.

NoMIRACL: hallucination and miss are separate

Status: Findings of EMNLP 2024.

NoMIRACL contains 56,057 contexts across 18 languages, ten language families, and 11 scripts. Hallucination rate is false-positive answers when no relevant passage exists; miss rate is failure to answer when evidence exists. Mixtral had the lowest reported mean hallucination at 17.4%; GPT-4 gave the best trade-off. Requiring explanations cut hallucination 9.7 points but increased misses 8.3. Always report both; it tests answer-versus-abstain, not answer correctness. EMNLP paper.

mtRAG and MTRAGEval: multi-turn compounding failures

Status: mtRAG TACL 2025; MTRAGEval SemEval 2026.

mtRAG has 110 human conversations, mean 7.7 turns and 842 tasks from Wikipedia, finance, government, and cloud, labeled answerable/partial/unanswerable/ conversational. Retrieval uses recall/nDCG@1/3/5/10; generation assesses faithfulness, appropriateness, naturalness, and completeness.

ELSER plus rewriting reported recall@10 0.64 and nDCG@10 0.54. Recall@5 fell from 0.89 on first turns to 0.47 later; the best automatic metric's Spearman against human win rate was only about 0.33. SemEval-2026's best retrieval nDCG@5 was 0.578, gold-context generation composite 0.783, and full-RAG 0.586, showing compounding errors. mtRAG and MTRAGEval.

GaRAGe: human labels expose weak factuality and deflection

Status: Findings of ACL 2025.

GaRAGe contains 2,366 questions and more than 35k human-annotated grounding passages from web and private documents, including dynamic facts and insufficient evidence. Tested frontier models reached at most 60% relevance-aware factuality, 31% true-positive deflection, and 58.9% F1 attribution to relevant sources. Time-sensitive and sparse private evidence was harder. It is strong evidence that citation-looking answers remain unreliable. ACL paper.

LongMemEval: retrieval is only one memory operation

Status: ICLR 2025 peer-reviewed.

LongMemEval has 500 questions over interactions totaling about 115k to 1.5M tokens and tests information extraction, cross-session reasoning, updates, temporal reasoning, and abstention. The key system lesson is that memory write policy, consolidation, overwrite/update semantics, and forgetting matter—not just ANN recall. ICLR paper.

TREC RAG: external adjudication and sentence-level citations

Status: annual NIST program, 2024–2026.

The track separates retrieval, generation with organizer-provided context, and full RAG on MS MARCO V2.1. Pooled relevance judgments, assessor-edited nuggets, nugget coverage, fluency, citation need, and sentence-level citation support provide stronger external adjudication than self-judged demos. Pin track year, corpus, topics, and judgments; pooled evaluation still has incompleteness. Official track and 2024 judgments.

Preprint-only evaluators

RAGBench remains a preprint: about 100k examples across 12 datasets with TRACe relevance/utilization/completeness/adherence, evaluator AUROC roughly 0.64–0.87, but human validation only 40 DelucionQA cases and many GPT-4 labels. Use only with this status caveat. Preprint.

3. A defensible product evaluation protocol

Build a stratified gold set

Start with 300–1,000 real queries, increasing for high-risk or heterogeneous products. Preserve natural frequency but oversample failure-critical slices:

• answerable, partially answerable, and unanswerable;
• popular and long-tail entities;
• fresh/dynamic and temporally scoped facts;
• single-hop, multi-hop, comparison, aggregation, and global synthesis;
• conflicts, counterfactuals, false premises, distractors, and duplicated text;
• long documents, tables, diagrams, scans, and layout;
• languages, scripts, and code-switching;
• first-turn, follow-up, correction, and topic-switch conversation;
• access-control roles/tenants and deleted content;
• malicious instructions, poisoned text, advertisements, and denial-of-service content.

Each example should have answerability, atomic reference claims, acceptable answers, evidence spans/source IDs, temporal validity, and authority labels. Double-label a subset, adjudicate disagreements, and report inter-annotator agreement.

Evaluate three layers

1. Retrieval: qrels Recall@\(k\), nDCG/MRR, context precision, claim recall, permission and temporal validity, ANN versus exact loss, latency/index cost.
2. Oracle-context generation: give gold evidence and measure claim P/R/F1, completeness, contradiction, citation entailment, abstention, and robustness to reordered/noisy/conflicting evidence. This isolates the reader.
3. End-to-end: all above plus answer utility, failure attribution, freshness, security, p95 latency, token/tool usage, and dollars per supported answer.

Use uncertainty and evaluator audits

• paired bootstrap or approximate randomization for per-query deltas;
• confidence intervals around aggregate rates; ARES/PPI when its assumptions and domain labels fit;
• tag/slice results before an overall mean;
• judge swaps and prompt sensitivity;
• 10–20% human audit of high-risk, system-disagreement, and judge-disagreement cases;
• multiple seeds for stochastic graph construction, retrieval agents, and generated query expansion.

Choose quality gates first, then plot a cost/latency Pareto frontier. A 0.5-point mean gain cannot compensate for unauthorized retrieval or a large high-stakes hallucination regression.

4. Freshness is an operational property

Retrieval makes updating possible; it does not make the index current.

FreshQA/FreshPrompt

Status: Findings of ACL 2024.

FreshQA includes dynamic and false-premise questions with more than 50k human judgments. FreshPrompt retrieves, curates, and organizes live search results and beats compared search-prompting/commercial systems in the paper; FreshEval is an autorater. The benchmark inevitably drifts. Store query time, pages, snippets, answer, and judgments; a score without its snapshot cannot be replayed. ACL paper.

Required controls

• change-data-capture and ingest SLA per source;
• event time and valid time, not only ingestion timestamp;
• version IDs/content hashes on every chunk and citation;
• temporal filtering, with decay only where newer is inherently better;
• explicit current-versus-historical query intent;
• conflicting-version detection and authority rules;
• index, summary, embedding, graph, and cache invalidation after corrections;
• deletion/unlearning propagation with verification;
• immutable snapshot replay for evaluation;
• monitoring for retrieval age, stale-answer rate, source lag, and temporal leakage.

5. Security: retrieval adds a hostile data plane

The threat model must include an attacker who can modify, publish, or trigger retrieval of content—not only an attacker who writes the user prompt.

AgentPoison

Status: NeurIPS 2024 peer-reviewed.

AgentPoison optimizes a trigger so triggered instructions land near poisoned memory embeddings. With below 0.1% poison rate it reports at least 80% attack success across driving, QA, and EHR agents with at most 1% benign degradation. The attack targets memory retrieval and downstream agent actions, showing that a tiny poisoned region can become a backdoor. NeurIPS paper.

PoisonedRAG

Status: USENIX Security 2025 peer-reviewed.

PoisonedRAG crafts retrieval-oriented and generation-oriented subtexts. Five malicious texts per target inserted among millions produced about 90% overall attack success; the Natural Questions example reached 97% in a database with 2,681,468 clean texts. Black- and white-box settings are considered. Paraphrasing and perplexity filters were insufficient. USENIX paper.

SafeRAG and “RAG LLMs are not safer”

SafeRAG (ACL 2025) is a manually constructed Chinese benchmark covering silver noise, inter-context conflict, soft advertisements, and white denial-of-service. Fourteen tested RAG components remained vulnerable and obvious attacks bypassed retrievers, filters, and LMs. Paper.

RAG LLMs Are Not Safer (NAACL 2025) evaluates 11 LMs and finds retrieval can reduce safety; even safe model plus nominally safe documents can yield unsafe output, and ordinary red-team attacks transfer poorly, requiring RAG-specific tests. Paper.

Defenses and their scope

• SeCon-RAG (NeurIPS 2025) filters with entity-intent-relation semantics and clustering, then checks answer/evidence conflicts. It is defense in depth, not a certificate. Paper.
• C-RAG (ICML 2024) applies conformal risk control, producing an upper confidence bound for a bounded generation-risk function under stated calibration/distribution-shift assumptions and sufficient conditions where RAG risk is below a vanilla LM. It certifies aggregate calibrated risk, not corpus provenance or prompt-injection safety. Paper.
• PRA-RAG (Findings ACL 2026) samples combinations of retrieved texts, identifies a robust subset from embedding geometry, aggregates a stable representation, and derives poisoned-content impact bounds. It reports attack success as low as 1% while retaining 71% accuracy across tested settings. Guarantees are threat-model-specific and add sampling/generation cost. Paper.
• Knowledgeable-R1 addresses misleading/counterfactual context but does not authenticate sources or block indirect instructions.

Required system mitigations

1. Treat retrieved bytes as untrusted data, never instructions; delimit and quote them in the prompt.
2. Preserve trust domain, signature, owner, tenant, ACL, source, version, and content hash before indexing.
3. Apply tenant/row access controls before ANN candidate exposure and again after reranking; do not rely on prompt instructions for authorization.
4. Sanitize active HTML/scripts/files and isolate evidence processing from tool/action privileges.
5. Use source allowlists/quarantine, duplicate and cluster analysis, conflict detection, and canary poison documents.
6. Monitor unusual retrieval neighborhoods, trigger-specific concentration, citation domains, tool actions, and answer/evidence divergence.
7. Keep immutable audit logs of query, candidates, scores, evidence hashes, model/prompt, actions, and response.
8. Test targeted poison, embedding backdoors, indirect prompt injection, conflicting sources, ads, DoS text, membership leakage, and exfiltration.
9. Separate the language model's read-only evidence role from any tool with network, filesystem, database, or messaging authority.

No perplexity, paraphrase, embedding-distance, or single LLM filter is a complete defense.

6. Citation implementation checklist

• Segment the answer into atomic claims before final rendering.
• Require every externally verifiable claim to map to an immutable source ID and exact span/page/region.
• Store the source version, retrieval timestamp, valid time, and content hash.
• Evaluate completeness separately from entailment and authority.
• Penalize “citation laundering”: citing a retrieved but non-supporting passage.
• Detect multiple citations that all derive from the same upstream source.
• Surface conflicts and temporal scope rather than silently selecting one.
• Verify links/objects remain accessible under the viewer's permissions.
• Human-review high-stakes health, legal, financial, safety, and operational actions.
• Retain the evidence shown to the model for audit even if the live source later changes, subject to retention and deletion law.

7. Minimum reproducibility manifest

experiment_id: immutable-id
query_set:
  version: ...
  hash: ...
  slice_schema: ...
corpus:
  sources: [...]
  snapshot_time: ...
  hash: ...
  permissions_version: ...
ingestion:
  parser: ...
  chunker: ...
  chunk_size: ...
  overlap: ...
retrieval:
  sparse_index: ...
  embedding_model_revision: ...
  ann_parameters: ...
  fusion: ...
  reranker_revision: ...
  candidate_k: ...
  evidence_budget_tokens: ...
generation:
  model_revision: ...
  prompt_hash: ...
  decoding: ...
evaluation:
  qrels_version: ...
  metrics: [...]
  judge_model_and_prompt: ...
  human_sample: ...
  confidence_interval: paired-bootstrap
trace_artifacts:
  retrieved_texts_scores_ranks: ...
  answer_claims_and_citations: ...
  latency_tokens_cost: ...

Without this manifest, a RAG result can be impossible to distinguish from corpus drift, a model API update, a changed top-\(k\), or evaluator drift.