Measuring the Invisible
The working page
This notebook evaluates retrieval, answer overlap, citation validity, lexical support, abstention, tagged slices, and uncertainty separately. It compares a sparse retrieve-then-generate baseline with the advanced modular pipeline.
The small teaching set demonstrates mechanics, not statistical claims about paper systems. A product evaluation needs hundreds or thousands of stratified, human-labeled queries and immutable corpus/model traces.
from pathlib import Path
import sys
ROOT = Path.cwd()
if ROOT.name == 'notebooks':
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / 'src'))
from rag_evolution.demo_data import demo_documents, demo_questions
from rag_evolution.evaluation import (
aggregate_metrics, evaluate_pipeline, evaluate_retriever, metrics_by_tag,
paired_bootstrap_delta, precision_at_k, recall_at_k, reciprocal_rank, ndcg_at_k,
)
from rag_evolution.pipeline import build_advanced_pipeline, build_baseline_pipeline
documents = demo_documents()
questions = demo_questions()
baseline = build_baseline_pipeline(documents)
advanced = build_advanced_pipeline(documents)
print(f'Evaluating {len(questions)} labeled examples over {len(documents)} documents')Evaluating 8 labeled examples over 14 documents
1. Verify metric behavior before trusting a harness
Metric unit tests catch denominator and cutoff mistakes. Retrieval precision, recall, MRR, and nDCG answer different questions and still do not prove that the generator used the evidence.
ranking = ['distractor', 'gold-a', 'gold-b']
gold = {'gold-a', 'gold-b'}
print(f'precision@2 = {precision_at_k(ranking, gold, 2):.3f}')
print(f'recall@2 = {recall_at_k(ranking, gold, 2):.3f}')
print(f'MRR = {reciprocal_rank(ranking, gold):.3f}')
print(f'nDCG@3 = {ndcg_at_k(ranking, gold, 3):.3f}')precision@2 = 0.500 recall@2 = 0.500 MRR = 0.500 nDCG@3 = 0.693
2. Retrieval layer: compare under one corpus and cutoff
The comparison holds documents, chunks, labels, and cutoff fixed. In a real study also hold ANN search depth, source filters, time, and reranker candidate budget fixed.
baseline_retrieval = evaluate_retriever(baseline, questions, k=3)
advanced_retrieval = evaluate_retriever(advanced, questions, k=3)
for name, rows in [('baseline', baseline_retrieval), ('advanced', advanced_retrieval)]:
mean = aggregate_metrics(rows)
print(f"{name:9s} recall@3={mean['recall@3']:.3f} MRR={mean['mrr']:.3f} nDCG@3={mean['ndcg@3']:.3f}")
print('\nPer-question recall deltas:')
for left, right in zip(baseline_retrieval, advanced_retrieval):
delta = right['recall@3'] - left['recall@3']
print(f"{left['id']:16s} baseline={left['recall@3']:.2f} advanced={right['recall@3']:.2f} delta={delta:+.2f}")baseline recall@3=0.938 MRR=1.000 nDCG@3=0.952 advanced recall@3=1.000 MRR=0.938 nDCG@3=0.954 Per-question recall deltas: q-dpr baseline=1.00 advanced=1.00 delta=+0.00 q-fid baseline=1.00 advanced=1.00 delta=+0.00 q-rag-dpr baseline=0.50 advanced=1.00 delta=+0.50 q-self-rag baseline=1.00 advanced=1.00 delta=+0.00 q-raptor baseline=1.00 advanced=1.00 delta=+0.00 q-visual baseline=1.00 advanced=1.00 delta=+0.00 q-long-context baseline=1.00 advanced=1.00 delta=+0.00 q-grip baseline=1.00 advanced=1.00 delta=+0.00
3. End-to-end layer: answers, citations, support, and abstention
answer_f1 measures overlap with a short reference and will undervalue a correct long extract. Citation precision/recall checks document labels. lexical_faithfulness is only a transparent overlap diagnostic—not semantic entailment. Human or calibrated claim-level judges are still required.
baseline_rows = evaluate_pipeline(baseline, questions)
advanced_rows = evaluate_pipeline(advanced, questions)
keys = ['recall@5', 'mrr', 'answer_f1', 'citation_precision', 'citation_recall', 'lexical_faithfulness', 'abstained']
print('metric baseline advanced')
print('-' * 48)
base_mean = aggregate_metrics(baseline_rows)
advanced_mean = aggregate_metrics(advanced_rows)
for key in keys:
print(f'{key:25s} {base_mean[key]:8.3f} {advanced_mean[key]:8.3f}')metric baseline advanced ------------------------------------------------ recall@5 1.000 1.000 mrr 1.000 0.938 answer_f1 0.275 0.275 citation_precision 0.875 0.875 citation_recall 1.000 1.000 lexical_faithfulness 1.000 1.000 abstained 0.000 0.000
4. Slice before averaging
An overall mean can hide that a technique helps comparisons but harms identifier lookup. Tags should reflect product risks: multi-hop, current, long-tail, unanswerable, multilingual, visual, conflict, and adversarial.
slices = metrics_by_tag(advanced_rows)
for tag in sorted(slices):
metrics = slices[tag]
print(f"{tag:14s} recall={metrics['recall@5']:.3f} citation_recall={metrics['citation_recall']:.3f} answer_f1={metrics['answer_f1']:.3f}")adaptive recall=1.000 citation_recall=1.000 answer_f1=0.424 agentic recall=1.000 citation_recall=1.000 answer_f1=0.049 architecture recall=1.000 citation_recall=1.000 answer_f1=0.391 comparison recall=1.000 citation_recall=1.000 answer_f1=0.286 hierarchical recall=1.000 citation_recall=1.000 answer_f1=0.323 lexical recall=1.000 citation_recall=1.000 answer_f1=0.485 multi-hop recall=1.000 citation_recall=1.000 answer_f1=0.286 multimodal recall=1.000 citation_recall=1.000 answer_f1=0.080 routing recall=1.000 citation_recall=1.000 answer_f1=0.162 single-hop recall=1.000 citation_recall=1.000 answer_f1=0.273
5. Pair systems and report uncertainty
Independent means throw away the pairing: both systems answer the same query. The paired bootstrap resamples query indices and estimates the mean delta interval. With only eight teaching questions the interval is illustrative and cannot support a serious conclusion.
advanced_recall = [row['recall@3'] for row in advanced_retrieval]
baseline_recall = [row['recall@3'] for row in baseline_retrieval]
delta, lower, upper = paired_bootstrap_delta(advanced_recall, baseline_recall, iterations=2000, seed=7)
print(f'advanced − baseline recall@3 = {delta:+.3f} (illustrative paired 95% interval {lower:+.3f}, {upper:+.3f})')advanced − baseline recall@3 = +0.062 (illustrative paired 95% interval +0.000, +0.188)
6. Inspect a multi-hop case causally
A final correct answer can hide an incomplete reasoning chain. Inspect candidates, selected contexts, answer claims, and citations for every failure—not only the final score.
example = next(item for item in questions if item.id == 'q-rag-dpr')
answer = advanced.ask(example.question)
print('QUESTION:', example.question)
print('GOLD DOCUMENTS:', example.relevant_document_ids)
print('PACKED DOCUMENTS:', tuple(result.chunk.document_id for result in answer.contexts))
print('CITED DOCUMENTS:', tuple(citation.document_id for citation in answer.citations))
print('ANSWER:', answer.text)QUESTION: What is the relationship and difference between DPR and the original RAG model?
GOLD DOCUMENTS: ('dpr-2020', 'rag-2020')
PACKED DOCUMENTS: ('rag-2020', 'dpr-2020', 'search-r1-2025', 'lara-2025', 'atlas-2022', 'self-rag-2023')
CITED DOCUMENTS: ('rag-2020', 'dpr-2020')
ANSWER: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks coupled a DPR question encoder with a frozen passage index and a BART generator. [rag-2020::c000] Dense Passage Retrieval (DPR) for Open-Domain Question Answering introduced a simple dual-encoder retriever trained with positive passages, in-batch negatives, and a hard BM25 negative. [dpr-2020::c000]
7. Test absence explicitly
NoMIRACL and sufficient-context work show that hallucination and miss rates trade off. Report both false answers when evidence is absent and unnecessary abstention when it is present.
absence_queries = [
'How are volcanic zircons dated with uranium lead ratios?',
'What dosage cures an imaginary disease called RAG fever?',
]
for query in absence_queries:
result = advanced.ask(query)
print(f'abstained={result.abstained!s:5s} confidence={result.confidence:.3f} | {query}')abstained=True confidence=0.000 | How are volcanic zircons dated with uranium lead ratios? abstained=False confidence=0.443 | What dosage cures an imaginary disease called RAG fever?
Production evaluation checklist
- Build 300–1,000+ stratified product queries with answerability, claims, exact evidence spans, time, authority, and permissions.
- Evaluate retrieval, oracle-context generation, and end-to-end behavior separately.
- Add conflict, counterfactual, noise, ordering, poison, indirect-instruction, deletion, and cross-tenant tests.
- Freeze corpus/query time, chunks, indexes, prompts, model/judge versions, seeds, and every retrieved text/score.
- Double-label and adjudicate a sample; audit automatic-judge disagreements and high-risk outputs.
- Apply hard safety/permission/freshness gates, then compare quality, p95 latency, index memory, tokens, tool calls, and dollars per supported answer on a Pareto frontier.