{ "cells": [ { "cell_type": "markdown", "metadata": { "tags": [ "field-notebook-cover", "field-notebook-style" ] }, "source": [ "\n", "\n", "
\n", "
LAB 03 · DIAGNOSIS
\n", "

Measuring the Invisible

\n", "
Tracing retrieval loss, context loss, and generation loss without hiding the cause
\n", "
an average cannot tell us where the evidence vanished
\n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The working page\n", "\n", "This notebook evaluates retrieval, answer overlap, citation validity, lexical support, abstention, tagged slices, and uncertainty separately. It compares a sparse retrieve-then-generate baseline with the advanced modular pipeline.\n", "\n", "The small teaching set demonstrates mechanics, not statistical claims about paper systems. A product evaluation needs hundreds or thousands of stratified, human-labeled queries and immutable corpus/model traces.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Evaluating 8 labeled examples over 14 documents\n" ] } ], "source": [ "from pathlib import Path\n", "import sys\n", "\n", "ROOT = Path.cwd()\n", "if ROOT.name == 'notebooks':\n", " ROOT = ROOT.parent\n", "sys.path.insert(0, str(ROOT / 'src'))\n", "\n", "from rag_evolution.demo_data import demo_documents, demo_questions\n", "from rag_evolution.evaluation import (\n", " aggregate_metrics, evaluate_pipeline, evaluate_retriever, metrics_by_tag,\n", " paired_bootstrap_delta, precision_at_k, recall_at_k, reciprocal_rank, ndcg_at_k,\n", ")\n", "from rag_evolution.pipeline import build_advanced_pipeline, build_baseline_pipeline\n", "\n", "documents = demo_documents()\n", "questions = demo_questions()\n", "baseline = build_baseline_pipeline(documents)\n", "advanced = build_advanced_pipeline(documents)\n", "print(f'Evaluating {len(questions)} labeled examples over {len(documents)} documents')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Verify metric behavior before trusting a harness\n", "\n", "Metric unit tests catch denominator and cutoff mistakes. Retrieval precision, recall, MRR, and nDCG answer different questions and still do not prove that the generator used the evidence." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "precision@2 = 0.500\n", "recall@2 = 0.500\n", "MRR = 0.500\n", "nDCG@3 = 0.693\n" ] } ], "source": [ "ranking = ['distractor', 'gold-a', 'gold-b']\n", "gold = {'gold-a', 'gold-b'}\n", "print(f'precision@2 = {precision_at_k(ranking, gold, 2):.3f}')\n", "print(f'recall@2 = {recall_at_k(ranking, gold, 2):.3f}')\n", "print(f'MRR = {reciprocal_rank(ranking, gold):.3f}')\n", "print(f'nDCG@3 = {ndcg_at_k(ranking, gold, 3):.3f}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Retrieval layer: compare under one corpus and cutoff\n", "\n", "The comparison holds documents, chunks, labels, and cutoff fixed. In a real study also hold ANN search depth, source filters, time, and reranker candidate budget fixed." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "baseline recall@3=0.938 MRR=1.000 nDCG@3=0.952\n", "advanced recall@3=1.000 MRR=0.938 nDCG@3=0.954\n", "\n", "Per-question recall deltas:\n", "q-dpr baseline=1.00 advanced=1.00 delta=+0.00\n", "q-fid baseline=1.00 advanced=1.00 delta=+0.00\n", "q-rag-dpr baseline=0.50 advanced=1.00 delta=+0.50\n", "q-self-rag baseline=1.00 advanced=1.00 delta=+0.00\n", "q-raptor baseline=1.00 advanced=1.00 delta=+0.00\n", "q-visual baseline=1.00 advanced=1.00 delta=+0.00\n", "q-long-context baseline=1.00 advanced=1.00 delta=+0.00\n", "q-grip baseline=1.00 advanced=1.00 delta=+0.00\n" ] } ], "source": [ "baseline_retrieval = evaluate_retriever(baseline, questions, k=3)\n", "advanced_retrieval = evaluate_retriever(advanced, questions, k=3)\n", "for name, rows in [('baseline', baseline_retrieval), ('advanced', advanced_retrieval)]:\n", " mean = aggregate_metrics(rows)\n", " print(f\"{name:9s} recall@3={mean['recall@3']:.3f} MRR={mean['mrr']:.3f} nDCG@3={mean['ndcg@3']:.3f}\")\n", "print('\\nPer-question recall deltas:')\n", "for left, right in zip(baseline_retrieval, advanced_retrieval):\n", " delta = right['recall@3'] - left['recall@3']\n", " print(f\"{left['id']:16s} baseline={left['recall@3']:.2f} advanced={right['recall@3']:.2f} delta={delta:+.2f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. End-to-end layer: answers, citations, support, and abstention\n", "\n", "`answer_f1` measures overlap with a short reference and will undervalue a correct long extract. Citation precision/recall checks document labels. `lexical_faithfulness` is only a transparent overlap diagnostic—not semantic entailment. Human or calibrated claim-level judges are still required." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "metric baseline advanced\n", "------------------------------------------------\n", "recall@5 1.000 1.000\n", "mrr 1.000 0.938\n", "answer_f1 0.275 0.275\n", "citation_precision 0.875 0.875\n", "citation_recall 1.000 1.000\n", "lexical_faithfulness 1.000 1.000\n", "abstained 0.000 0.000\n" ] } ], "source": [ "baseline_rows = evaluate_pipeline(baseline, questions)\n", "advanced_rows = evaluate_pipeline(advanced, questions)\n", "keys = ['recall@5', 'mrr', 'answer_f1', 'citation_precision', 'citation_recall', 'lexical_faithfulness', 'abstained']\n", "print('metric baseline advanced')\n", "print('-' * 48)\n", "base_mean = aggregate_metrics(baseline_rows)\n", "advanced_mean = aggregate_metrics(advanced_rows)\n", "for key in keys:\n", " print(f'{key:25s} {base_mean[key]:8.3f} {advanced_mean[key]:8.3f}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Slice before averaging\n", "\n", "An overall mean can hide that a technique helps comparisons but harms identifier lookup. Tags should reflect product risks: multi-hop, current, long-tail, unanswerable, multilingual, visual, conflict, and adversarial." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "adaptive recall=1.000 citation_recall=1.000 answer_f1=0.424\n", "agentic recall=1.000 citation_recall=1.000 answer_f1=0.049\n", "architecture recall=1.000 citation_recall=1.000 answer_f1=0.391\n", "comparison recall=1.000 citation_recall=1.000 answer_f1=0.286\n", "hierarchical recall=1.000 citation_recall=1.000 answer_f1=0.323\n", "lexical recall=1.000 citation_recall=1.000 answer_f1=0.485\n", "multi-hop recall=1.000 citation_recall=1.000 answer_f1=0.286\n", "multimodal recall=1.000 citation_recall=1.000 answer_f1=0.080\n", "routing recall=1.000 citation_recall=1.000 answer_f1=0.162\n", "single-hop recall=1.000 citation_recall=1.000 answer_f1=0.273\n" ] } ], "source": [ "slices = metrics_by_tag(advanced_rows)\n", "for tag in sorted(slices):\n", " metrics = slices[tag]\n", " print(f\"{tag:14s} recall={metrics['recall@5']:.3f} citation_recall={metrics['citation_recall']:.3f} answer_f1={metrics['answer_f1']:.3f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Pair systems and report uncertainty\n", "\n", "Independent means throw away the pairing: both systems answer the same query. The paired bootstrap resamples query indices and estimates the mean delta interval. With only eight teaching questions the interval is illustrative and cannot support a serious conclusion." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "advanced − baseline recall@3 = +0.062 (illustrative paired 95% interval +0.000, +0.188)\n" ] } ], "source": [ "advanced_recall = [row['recall@3'] for row in advanced_retrieval]\n", "baseline_recall = [row['recall@3'] for row in baseline_retrieval]\n", "delta, lower, upper = paired_bootstrap_delta(advanced_recall, baseline_recall, iterations=2000, seed=7)\n", "print(f'advanced − baseline recall@3 = {delta:+.3f} (illustrative paired 95% interval {lower:+.3f}, {upper:+.3f})')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Inspect a multi-hop case causally\n", "\n", "A final correct answer can hide an incomplete reasoning chain. Inspect candidates, selected contexts, answer claims, and citations for every failure—not only the final score." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "QUESTION: What is the relationship and difference between DPR and the original RAG model?\n", "GOLD DOCUMENTS: ('dpr-2020', 'rag-2020')\n", "PACKED DOCUMENTS: ('rag-2020', 'dpr-2020', 'search-r1-2025', 'lara-2025', 'atlas-2022', 'self-rag-2023')\n", "CITED DOCUMENTS: ('rag-2020', 'dpr-2020')\n", "ANSWER: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks coupled a DPR question encoder with a frozen passage index and a BART generator. [rag-2020::c000] Dense Passage Retrieval (DPR) for Open-Domain Question Answering introduced a simple dual-encoder retriever trained with positive passages, in-batch negatives, and a hard BM25 negative. [dpr-2020::c000]\n" ] } ], "source": [ "example = next(item for item in questions if item.id == 'q-rag-dpr')\n", "answer = advanced.ask(example.question)\n", "print('QUESTION:', example.question)\n", "print('GOLD DOCUMENTS:', example.relevant_document_ids)\n", "print('PACKED DOCUMENTS:', tuple(result.chunk.document_id for result in answer.contexts))\n", "print('CITED DOCUMENTS:', tuple(citation.document_id for citation in answer.citations))\n", "print('ANSWER:', answer.text)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Test absence explicitly\n", "\n", "NoMIRACL and sufficient-context work show that hallucination and miss rates trade off. Report both false answers when evidence is absent and unnecessary abstention when it is present." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "abstained=True confidence=0.000 | How are volcanic zircons dated with uranium lead ratios?\n", "abstained=False confidence=0.443 | What dosage cures an imaginary disease called RAG fever?\n" ] } ], "source": [ "absence_queries = [\n", " 'How are volcanic zircons dated with uranium lead ratios?',\n", " 'What dosage cures an imaginary disease called RAG fever?',\n", "]\n", "for query in absence_queries:\n", " result = advanced.ask(query)\n", " print(f'abstained={result.abstained!s:5s} confidence={result.confidence:.3f} | {query}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Production evaluation checklist\n", "\n", "1. Build 300–1,000+ stratified product queries with answerability, claims, exact evidence spans, time, authority, and permissions.\n", "2. Evaluate retrieval, oracle-context generation, and end-to-end behavior separately.\n", "3. Add conflict, counterfactual, noise, ordering, poison, indirect-instruction, deletion, and cross-tenant tests.\n", "4. Freeze corpus/query time, chunks, indexes, prompts, model/judge versions, seeds, and every retrieved text/score.\n", "5. Double-label and adjudicate a sample; audit automatic-judge disagreements and high-risk outputs.\n", "6. Apply hard safety/permission/freshness gates, then compare quality, p95 latency, index memory, tokens, tool calls, and dollars per supported answer on a Pareto frontier." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.9" }, "rag_evolution": { "presentation": "plain-field-notebook", "visual_version": 2 } }, "nbformat": 4, "nbformat_minor": 5 }