A RAG answer is wrong in production. Which experiment tells us whether the corpus, retrieval, reading, citation, or release process failed?
# Measuring the Answering Machine
The seductive RAG demo has one input and one output. A question is typed; a fluent, cited answer appears. Evaluation begins by refusing that visual simplicity. Between request and prose lies a chain of contingent events: the needed fact must exist in an allowed corpus, survive parsing and chunking, enter an index, be reached by a query, rank above distractors, survive reranking and context packing, be interpreted under the correct time and authority, become an answer claim, and receive a citation that actually supports it. A single end-to-end score compresses all of these events into a verdict and destroys the diagnosis.
The unit worth following is the claim. Let a generated answer contain atomic claims \(A=\{a_1,\ldots,a_m\}\), and let a reference describe required claims \(Y=\{y_1,\ldots,y_n\}\). We need two relations that are often blurred together. Let \(C(a)\) mean that claim \(a\) is correct under the chosen reference or adjudicated world state, and let \(S(a)\) mean that the supplied context entails it. Correctness and contextual support must first be reported separately:
\[
P_{correct}=\frac{\sum_{a\in A}C(a)}{|A|},\qquad
F_{context}=\frac{\sum_{a\in A}S(a)}{|A|}.
\]
A claim counts as **grounded-correct** only at their intersection. Completeness then asks which required claims were recovered correctly, while grounded precision asks how much generated material was both true and supported:
\[
P_{grounded}=\frac{\sum_{a\in A}C(a)S(a)}{|A|},\qquad
R_{required}=\frac{|Y_{\mathrm{covered\ correctly}}|}{|Y|},\qquad
F_{1,grounded}=\frac{2P_{grounded}R_{required}}{P_{grounded}+R_{required}}.
\]
Even this intersection should not erase its components. A faithfully repeated poison passage has high \(F_{context}\) and low \(P_{correct}\); a true statement supplied from parametric memory may have the reverse pattern under a strict context-only policy. A one-sentence answer may have perfect grounded precision and omit the decision-critical exception. A long answer may achieve high recall by adding unsupported material. Exact match is still appropriate for a code, date, or short entity, and ROUGE may describe surface overlap, but neither is a grounding argument. The claim ledger is what allows retrieval evidence, answer content, and citations to meet at the same granularity.
## The three-room experiment
Every serious evaluation should contain three rooms with doors that can be closed independently. In the first room, the generator is absent and retrieval is tested against evidence judgments. In the second, retrieval is replaced with gold evidence and the generator is tested as a reader. In the third, the complete system runs under the permissions, time, latency, and cost constraints of production.
The retrieval room begins with familiar information-retrieval measures. For a relevance set \(G_q\), retrieved list \(R_q\), and cutoff \(k\),
\[
P@k=\frac{|R_{q,1:k}\cap G_q|}{k},\qquad
R@k=\frac{|R_{q,1:k}\cap G_q|}{|G_q|}.
\]
MRR rewards the rank of the first relevant item. nDCG rewards graded relevance near the top:
\[
DCG@k=\sum_{i=1}^{k}\frac{2^{g_i}-1}{\log_2(i+1)},\qquad
nDCG@k=\frac{DCG@k}{IDCG@k}.
\]
These values are properties of a system *and its qrels*. An “answer-containing” passage may contradict the answer while repeating its string. A pooled judgment set may omit a relevant passage found only by a new retriever. Page labels do not necessarily validate a chunk. For RAG, retrieval reporting should therefore add claim recall, context precision, authority, temporal validity, permission validity, ANN-versus-exact loss, and coverage under a fixed evidence-token budget. The first room asks: did usable evidence become available to the reader, not merely did a familiar document identifier appear?
The oracle-context room supplies the exact supporting spans and deliberately removes the retriever's excuse. It measures claim precision and completeness, contradictions, robustness to context order, citation placement and entailment, and behavior when the gold context is insufficient. Distractor experiments then add irrelevant, duplicated, stale, and counterfactual passages in controlled amounts. If the reader fails with clean gold evidence, changing embeddings will not repair it. If it succeeds on gold but fails end to end, the failure lies upstream or in context selection.
The third room restores the actual corpus snapshot, ACLs, chunker, retriever, reranker, context builder, generator, verifier, caches, and deadlines. It adds task utility, freshness, security, p95 and p99 latency, cost, energy, fallback behavior, and provenance. A clean experimental record retains every candidate ID and score, the exact evidence shown, source version and valid time, prompt and model revisions, answer claims and citations, stage timings, retries, and the resolved configuration. Without this trace, an end-to-end miss is a story, not a diagnosis.
Four-way isolation. For each release candidate, run closed-book, gold-context, retrieved-context, and retrieved-context-with-distractors conditions. Hold the generator fixed while changing retrieval, then hold evidence fixed while changing the generator. Report per-query deltas rather than four unrelated means. The crossed design exposes cases in which a stronger model merely masks weaker retrieval.
## Evaluators are instruments, not oracles
The appeal of automatic RAG evaluation is obvious: reference answers and human claim labels are expensive, while an LLM can split prose into claims and judge support at scale. The danger is equally plain. An evaluator is another model with a prompt, training distribution, context limit, failure modes, and version lifecycle. It must be calibrated like a measurement instrument.
[RAGAS](https://aclanthology.org/2024.eacl-demo.16/) provided a practical vocabulary for early diagnosis. Its original faithfulness metric extracts answer statements and computes the fraction supported by context. Answer relevance generates possible questions from the answer and averages their embedding similarity to the original question. Context relevance measures the fraction of context sentences judged useful. On the 50-question synthetic WikiEval set, reported agreement with humans was .95 for faithfulness, .78 for answer relevance, and .70 for context relevance, compared with .72/.52/.63 for GPT-score. Those results justify a useful smoke detector, not a universal ruler: the validation set was tiny, answer relevance does not test factuality, and the package's models and metrics have continued to evolve. A reproducible run pins library, judge, embeddings, prompts, temperature, and parsing behavior.
[ARES](https://aclanthology.org/2024.naacl-long.20/) approaches the calibration problem more deliberately. It synthesizes in-domain examples, trains DeBERTa-v3-Large judges for context relevance, answer faithfulness, and answer relevance, and corrects large-scale model predictions with a labeled sample using prediction-powered inference. In simplified form,
\[
\hat\mu_{PPI}=
\frac{1}{N}\sum_{i=1}^{N}f(x_i)
+\frac{1}{n}\sum_{j=1}^{n}\bigl(y_j-f(x_j)\bigr).
\]
The first term supplies scale; the labeled residual term corrects bias and supports an interval. ARES asks for at least five in-domain demonstrations and roughly 150 labeled examples, with experiments commonly using 300. Across its KILT, SuperGLUE, and AIS tasks, it reported evaluator-accuracy improvements over RAGAS of 59.3 points for context relevance and 14.4 for answer relevance; aggregate hallucination estimates were within 2.5 points while using 78% fewer annotations. The important qualification is “aggregate.” PPI can estimate a system rate well while an individual judge label remains wrong. ARES is attractive for a stable domain and recurring release decision, less so as an instant per-answer truth machine.
[RAGChecker](https://proceedings.neurips.cc/paper_files/paper/2024/file/27245589131d17368cccdfa990cbf16e-Paper-Datasets_and_Benchmarks_Track.pdf) makes the claim path explicit. Its 4,162-question benchmark spans ten English domains and reports retriever claim recall and context precision; generator faithfulness and context utilization; relevant- and irrelevant-noise sensitivity; hallucination; correct unsupported self-knowledge; and overall claim precision, recall, and F1. Its reported Pearson/Spearman human correlation was .6193/.6090, versus .4831/.5723 for the strongest compared RAGAS answer-similarity measure. Increasing retrieved chunks from five to twenty raised claim recall from 61.5 to 77.6 while also increasing noise sensitivity. That is exactly the trade-off a single retrieval metric hides. RAGChecker is a strong diagnostic, but claim extraction and entailment remain expensive, model-dependent operations. Disputed and high-risk cases still need human review.
[RAGTruth](https://aclanthology.org/2024.acl-long.585/) is best read as a detector laboratory. It contains 2,965 prompts and 17,790 responses from six 2023-era models across QA, data-to-text, and summarization, with 14,289 annotated hallucination spans. Of its responses, 7,664—43.1%—contain at least one hallucination. A fine-tuned Llama-2-13B detector reached response-level F1 78.7, but span F1 only 52.7; prompted GPT-4 reached 63.4 and 28.3. Span localization is therefore much harder than declaring a response suspicious. RAGTruth evaluates reference-grounding detection, not whether retrieval found the right evidence, and its strict policy may label a true outside fact unsupported. The product must decide whether its contract is context-only grounding or permissive external knowledge before adopting the labels.
A release gate should never be “RAGAS increased.” It should name the metric, judge, calibration set, interval, product slice, and the failure rate that the change is allowed to trade away.
## Benchmarks as stress chambers
Public benchmarks are most useful as stress chambers. We do not move into one and declare it the world; we expose a system to a named force and observe how it bends. A suite built for zero-shot retrieval cannot establish citation faithfulness. A reader test with supplied passages cannot diagnose the index. A frozen Wikipedia snapshot cannot represent a policy that changed this morning. The benchmark belongs in the evaluation only when its force resembles a product risk.
[BEIR](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/65b9eea6e1cc6bb9f0cd2a47751a186f-Abstract-round2.html) is the useful cold room for retrieval transfer. Across eighteen domains, the original work found BM25 stubbornly competitive and showed that a sparse first stage plus cross-encoder reranking often beat either fashion or simplicity alone. Additional TREC-COVID judgments materially changed ANCE's apparent performance, revealing that qrels are historical pools rather than complete truth. BEIR can tell us whether a retriever travels; it cannot tell us whether a generated claim is grounded.
The [Comprehensive RAG Benchmark](https://proceedings.neurips.cc/paper_files/paper/2024/hash/1435d2d0fca85a84d83ddcb754f58c29-Abstract-Datasets_and_Benchmarks_Track.html) is a weather chamber. Its questions vary in popularity, complexity, and dynamism and can draw on web pages, a knowledge graph, and mock APIs. More retrieval improved reported accuracy while also leaving substantial hallucination, so accurate, missing, and incorrect answers must remain separate. Its enduring lesson is that an index does not become fresh merely because the architecture has a retriever; preserve query time, source snapshot, and API state.
Conversation and abstention need different rooms. [mtRAG](https://aclanthology.org/2025.tacl-1.36/) makes later turns carry unresolved references and hidden conversational state; retrieval recall falls sharply after the first turn, a failure a standalone QA set cannot see. [NoMIRACL](https://aclanthology.org/2024.findings-emnlp.730/) separates false answering when no relevant passage exists from missing an answer when evidence is present across eighteen languages. Improving one by refusing more often can worsen the other. Both teach the same experimental habit: preserve the two sides of a trade rather than celebrating the side a prompt happened to optimize.
The atlas behind these folios keeps the fuller instrument cabinet: KILT for provenance-gated history, MTEB and MMTEB for representation breadth, RGB for noise and counterfactual context, the peer-reviewed [CRUD-RAG](https://doi.org/10.1145/3701228) benchmark for create/read/update/delete operations, BRIGHT for reasoning-intensive retrieval, and TREC RAG for shared citation adjudication. They are chosen by risk and pinned by version—not accumulated into one ceremonial average.
## Citations and abstention are paired controls
A citation is not one binary property. For every externally verifiable claim, ask whether a citation is present, whether the cited span entails the claim, whether its source is authoritative, whether the source version is authentic and accessible to this viewer, and whether the evidence causally influenced the answer. A model-generated URL that was never retrieved fails before entailment is considered. Several citations copied from one upstream article are not independent corroboration. A correct claim with an irrelevant citation is citation laundering.
Implementation should constrain the generator to source IDs supplied by the system, resolve them to immutable versions and exact page, span, table cell, or image region, store content hashes and retrieval time, and evaluate completeness separately from support and authority. The [TREC RAG track](https://trec-rag.github.io/) is valuable because it separates retrieval, organizer-context generation, and full RAG and adjudicates answer nuggets, citation need, and sentence-level support. Like any annual program, it must be identified by year, corpus, topics, and judgment release.
Abstention controls what happens when citation cannot be made honestly. It has two costs: a false answer when evidence is absent, and an unnecessary refusal when evidence is sufficient. If a system answers only easy queries, accuracy can rise while utility collapses. Report risk against coverage, plus the false-answer and unnecessary-abstention rates. Calibrate by domain, source authority, freshness, and consequence. The threshold for a restaurant recommendation need not equal the threshold for a drug interaction.
Would you rather deploy a system with 92% accuracy at 40% answer coverage, or 86% accuracy at 90% coverage?
The question has no context-free answer. Plot the curve, price both error types, and make the operating point a declared product decision rather than a hidden prompt side effect.
## Calibration, uncertainty, and experimental honesty
A credible comparison is paired: the old and new systems answer the same examples, and the analysis resamples query-level differences. Paired bootstrap intervals or approximate randomization preserve this structure. Report effect sizes and intervals, not only a significance label. Use multiple seeds for stochastic graph construction, query expansion, agent search, and sampling. Slice results before averaging: answerability, temporal class, domain, language, document form, hop count, authority, user role, and risk tier often move in opposite directions.
Judge calibration deserves its own experiment. Double-label a stratified subset, adjudicate disagreement, report human-human agreement, compare judge prompts and model revisions, and audit perhaps 10–20% of high-risk, system-disagreement, and judge-disagreement cases. Keep evaluator training, threshold tuning, and final testing separate. A dynamic benchmark needs an immutable query time, source snapshot, and answer snapshot. A public benchmark exposed for years also needs contamination analysis or a temporal/private holdout.
Do not optimize twenty metrics until one happens to improve. Pre-register primary gates and acceptable regressions. ACL violations, unsupported high-stakes claims, invalid citations, and deletion failures are usually hard gates. Among systems that pass, compare a Pareto surface of quality, latency, cost, and energy. If a scalar is necessary for automation, publish its weights and retain the component dashboard.
## The system under load
Offline quality without a serving budget is an incomplete result. Measure p50, p95, and p99 separately for authentication and policy, query planning, each retriever, fan-out joins, reranking, context construction, time to first token, decoding, tools, verification, and end to end. Record cold and warm cache, concurrency, filter selectivity, context length, output length, timeout, retry, and fallback. Iterative agents can have modest mean latency and catastrophic tails because each search step is sequential.
Per-request economic cost can be written as
\[
C=C_{embed}+C_{search}+C_{rerank}+C_{prompt}+C_{decode}+C_{tools}+C_{verify}+C_{network}.
\]
Amortized cost adds parsing, OCR, embeddings, graph or summary construction, index builds, replicas, backups, evaluation, and human review. The meaningful denominator is not attempts but correct, sufficiently supported answers—or resolved user tasks. Report cost per request, per answered request, and per correct cited answer. Energy belongs beside money: record CPU- and GPU-seconds, device power or measured joules where available, index-build and ingestion energy, and kWh per thousand representative queries. Carbon claims require region- and time-specific electricity assumptions; model tokens are not an energy unit. A graph that saves generation tokens after an enormous recurring rebuild may move cost rather than reduce it.
## The queue, the shard, and the stale replica
A latency percentile is an observation, not a capacity plan. The same pipeline that answers beautifully at one request per second may collapse at one hundred because its stages do not saturate together. Dense search may be memory-bandwidth bound, a cross-encoder may be GPU-batch bound, a graph traversal may be dominated by irregular reads, and generation may hold scarce accelerator memory for seconds. Arrival bursts create queues between those stages. Once a queue grows, the request reaching the model is already old; a retry can double the work precisely when the system has the least room to perform it.
Capacity testing should therefore vary concurrency, arrival shape, query class, filter selectivity, evidence depth, context length, output length, cache warmth, and agent-step count. Plot achieved throughput beside queueing time and utilization for every constrained pool. Find the knee where a small increase in arrivals produces a large increase in tail latency. Little's law, \(L=\lambda W\), is a useful accounting identity: if throughput \(\lambda\) remains fixed while time in the system \(W\) rises, work in flight \(L\) must accumulate somewhere. The trace should make that somewhere visible.
Batching helps only when its waiting policy respects deadlines. Embedding and reranking requests often benefit from dynamic microbatches, while autoregressive generation can use continuous batching. Yet a low-latency query should not wait behind a large batch assembled for throughput, and one tenant should not fill an accelerator queue at the expense of another. A scheduler needs maximum wait, batch and token budgets, admission priorities, per-tenant fairness, and cancellation that actually releases downstream work. Deadline propagation matters more than independent timeouts: a reranker with 80 milliseconds left should not begin a 200-millisecond job merely because its local timeout is one second.
At overload, **backpressure** is an answer. Bound every queue, reject or shed low-priority work before expensive fan-out, and preserve capacity for authorization, deletion, and other safety-critical paths. Retries need budgets, exponential backoff, jitter, idempotency, and a distinction between a transient failure and a request that is intrinsically too expensive. Circuit breakers isolate an unhealthy embedding service, search shard, model provider, or tool before its latency infects the whole graph. Bulkheads keep separate tenants and workload classes from exhausting the same pool. A degraded mode might reduce candidate depth, skip a nonessential reranker, route to a smaller generator, return extractive evidence, or abstain. It must never weaken ACLs, provenance validation, deletion semantics, or high-risk citation gates merely to remain available.
Sharding introduces a different family of losses. A corpus can be partitioned by tenant, source, language, time, or a hash of the document identity; vectors can also be distributed by index-specific partitions. Each choice changes fan-out and failure behavior. Tenant sharding strengthens isolation but may create hot or tiny shards. Hash sharding balances documents but requires broad query fan-out. Semantic partitions reduce search breadth and risk routing misses. Time partitions make freshness and archival policy explicit but complicate queries that cross versions. Measure shard skew in document count, vector bytes, update rate, query rate, filter selectivity, and latency—not only total storage.
A coordinator must merge partial rankings without silently treating a missing shard as an empty result. It should carry shard generation, timeout, truncation, and failure metadata into the trace and, when they matter, into the answer policy. Approximate indexes deserve recall audits per shard and filter slice, because an aggregate sample can hide a damaged or under-probed partition. Rebalancing must preserve stable document identities and avoid serving the same item twice or not at all while ownership moves.
Replicas make search available and time ambiguous. A newly ingested correction may be visible on one replica while an older answer remains cached or searchable on another. The system needs a declared consistency contract: which operations require read-after-write, how an index generation becomes active, whether a query may mix generations across shards, and what maximum replication lag is acceptable for each source class. Release manifests should identify the corpus and index generation actually queried, not the generation the control plane intended to deploy. Cache keys must include every value that changes evidence eligibility—tenant and role policy, corpus/index generation, query transformation, temporal cutoff, retriever and reranker versions, and context policy—otherwise a fast response may be an answer from the wrong world.
Multi-region operation turns these choices into recovery policy. Decide where source-of-truth ingestion occurs, how manifests and tombstones replicate, which indexes can be rebuilt versus restored, how region failover preserves authorization keys and valid time, and whether an isolated region is allowed to serve stale evidence. Recovery-point and recovery-time objectives should be tested by losing a region, a shard generation, a queue, and a provider—not inferred from the existence of backups. Restore drills must verify query behavior and deletions after recovery; a successfully copied index that resurrects a removed document is not a successful restore.
Production readiness is not “the service has replicas.” It is knowing what a partial replica, a mixed index generation, a full queue, and a failed dependency mean for the truth conditions of the answer.
## Observability is evaluation in motion
Production traces should mirror the experimental decomposition. A request receives a stable ID and an absolute deadline. Events record policy resolution, classification and route, transformed queries, retrieval starts and completions, fusion and reranking, evidence selection and compression, generation and first token, verification, response, and feedback. Each event carries version IDs, counts, latency, cost, opaque evidence IDs, decision reasons, and error state. Content is minimized, redacted, tenant-partitioned, retained briefly, and accessible only for its declared purpose; observability data is itself a sensitive corpus.
Online signals begin upstream. Monitor connector lag, parser and OCR confidence, chunk and vector counts, embedding norms, duplication, index generation, ANN recall samples, tombstone backlog, and language or document-type shift. At retrieval, observe no-result rates, score distributions, sparse/dense overlap, candidate-to-selected-to-cited survival, source concentration, filters, loops, and fallback. At generation, observe answer/partial/abstain/error, claim and citation counts, invalid IDs, deterministic date and number consistency, sampled support, schema failure, and conflict disclosure. At the system level, observe saturation, cache correctness, retries, cost per supported answer, energy budget, and error-budget burn.
Most of these are proxies because production truth arrives late. Sentinel questions with known evidence, user corrections, escalations, and periodic human audits help, but none should train and evaluate the same judge on the same feedback. Maintain a rolling adjudicated set and a separate immutable regression set. When model providers, embeddings, tokenizers, parsers, or corpus composition change, calibration can drift even if the product version label does not.
## Release is an experiment with an exit
A release begins offline with a frozen corpus, query set, qrels, exact configuration, per-query traces, paired intervals, adversarial suites, and human review of important disagreements. It proceeds to shadow traffic, where the candidate observes production-shaped requests without serving its output. Shadowing reveals routes, candidate sets, latency, and safety differences, but not the user's reaction to the unseen answer. A canary then serves a small, stably assigned, representative population. Expansion is conditional on hard gates and slice-specific error budgets. The rollback target is an immutable model, prompt, index, policy, and cache-compatible generation—not merely yesterday's code commit.
Every behavior-changing parameter should be versioned: corpus and ACL generation, parser and chunker, analyzers and embeddings, ANN settings, fusion depths, reranker, router, context order and budget, prompt and decoding, judge and threshold, caches, timeouts, fallback, and tool scopes. A release manifest that says “GPT-4 plus vector search” cannot reproduce a request.
When an incident occurs, preserve the minimum authorized forensic trace and freeze the affected versions. Classify the symptom before tuning: retrieval-quality incidents begin with source counts, parser failures, index generation, ANN recall, ACL filters, score shifts, and reranking; freshness incidents begin with connector watermarks, index lag, replicas, temporal metadata, and caches; grounding incidents begin with evidence survival, prompt/context order, claim-to-citation mapping, and verifier changes; latency incidents begin with stage saturation, candidate and token growth, loops, retries, and dependencies. Security incidents require containing the source, index, tool, or tenant route; quarantining malicious content; rebuilding a clean generation; invalidating caches; rotating exposed credentials; and replaying targeted tests before gradual restoration.
Release rehearsal. Before canary, inject one parser regression, one stale replica, one missing ACL, one counterfeit citation, one ten-step agent loop, and one slow reranker into a staging generation. Verify that the trace localizes each fault, alerts name the affected slice, degraded mode preserves security and citation gates, and rollback atomically restores compatible encoder, index, prompt, policy, and caches.
The mature evaluation question is not “Which RAG score is best?” It is “Which evidence chain changed, how certain are we that the change is real, what did it cost under load, whom could it harm, and can we reverse it?” Claim-level accounting answers the first part. Layered experiments locate causality. Calibrated judges and human labels quantify uncertainty. Production traces reveal drift. Staged rollout limits exposure. Incident rehearsal ensures that measurement still matters when the notebook's clean assumptions meet a mutable world.