{ "cells": [ { "cell_type": "markdown", "metadata": { "tags": [ "field-notebook-cover", "field-notebook-style" ] }, "source": [ "\n", "\n", "
\n", "
LAB 05 · LEARNING
\n", "

Teaching Search to Choose

\n", "
Negatives, objectives, transformations, fusion, reranking, selection, and reward
\n", "
relevance is only useful when it survives downstream
\n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The working page\n", "\n", "This lab follows the learning signal through a modern retrieval stack. It\n", "treats negatives, score calibration, reranking, evidence-set selection, and\n", "retrieval-control rewards as first-class experimental variables.\n", "\n", "**Learning outcomes**\n", "\n", "- compute InfoNCE, pairwise, listwise, distillation, DPO, and policy-gradient objectives;\n", "- identify false negatives and label leakage in hard-negative mining;\n", "- compare score fusion with rank fusion;\n", "- distinguish candidate ranking from budgeted evidence coverage;\n", "- attribute relevant evidence lost at retrieval, reranking, or packing.\n", "\n", "Companion chapters: [Retrieval and ranking](../research/retrieval_and_ranking.md)\n", "and [Training and optimization](../research/training_and_optimization.md).\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Repository root: resolved from the notebook location\n" ] } ], "source": [ "from pathlib import Path\n", "import sys\n", "\n", "ROOT = Path.cwd()\n", "if not (ROOT / \"src\").exists():\n", " ROOT = ROOT.parent\n", "sys.path.insert(0, str(ROOT / \"src\"))\n", "print(\"Repository root: resolved from the notebook location\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Labels define what “relevant” means\n", "\n", "Positives may be human qrels, answer-containing passages, cited sources,\n", "supporting facts, clicked documents, successful tool results, synthetic\n", "teacher labels, or passages that improve a downstream reader. These signals\n", "disagree. Answer string containment can reward a passage that repeats a\n", "false claim; clicks encode position bias; citations may be incomplete;\n", "teacher labels inherit model bias; downstream utility can reward spurious\n", "shortcuts.\n", "\n", "Preserve label provenance and uncertainty. Split by source/time/template to\n", "prevent leakage. Evaluate retriever recall, reader robustness, and generator\n", "parametric knowledge separately.\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "temperature 0.25 P(positive) 0.9734 loss 0.027\n", "temperature 0.5 P(positive) 0.8532 loss 0.1587\n", "temperature 1.0 P(positive) 0.6717 loss 0.3979\n", "temperature 2.0 P(positive) 0.5197 loss 0.6545\n", "Hinge easy/hard: 0.0 2.5\n" ] } ], "source": [ "from rag_evolution.training import contrastive_loss, pairwise_hinge_loss, softmax\n", "\n", "for temperature in (0.25, 0.5, 1.0, 2.0):\n", " probabilities = softmax((3.0, 2.1, 0.5), temperature)\n", " loss = contrastive_loss(3.0, (2.1, 0.5), temperature)\n", " print(\"temperature\", temperature, \"P(positive)\", round(probabilities[0], 4), \"loss\", round(loss, 4))\n", "print(\"Hinge easy/hard:\", pairwise_hinge_loss(3.0, 1.0), pairwise_hinge_loss(1.0, 2.5))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Contrastive learning is largely a negative-sampling design\n", "\n", "For query (q_i), positive (d_i^+), negatives (d_j^-), a common loss is\n", "\n", "\\[\n", "-\\log\\frac{\\exp(s(q_i,d_i^+)/\\tau)}\n", "{\\exp(s(q_i,d_i^+)/\\tau)+\\sum_j\\exp(s(q_i,d_j^-)/\\tau)}.\n", "\\]\n", "\n", "In-batch negatives are cheap but may contain alternate positives. BM25/dense\n", "hard negatives teach fine distinctions but can concentrate annotation\n", "errors. Cross-encoder mining adds teacher bias; same-source negatives may be\n", "genuinely supportive. Track source identity, answer aliases, qrels, and\n", "teacher relevance, and quarantine candidates that may be false negatives.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Mask: ((False, False, True), (True, False, True))\n", "Unmasked loss: 0.3679\n", "False-negative-aware loss: 0.0796\n" ] } ], "source": [ "from rag_evolution.training import false_negative_mask, in_batch_contrastive_loss\n", "\n", "similarities = (\n", " (3.2, 3.0, 0.2), # document 1 is an unlabeled alternate positive for query 0\n", " (0.1, 3.1, 0.4),\n", ")\n", "mask = false_negative_mask(({\"doc-0\", \"doc-1\"}, {\"doc-1\"}), (\"doc-0\", \"doc-1\", \"doc-2\"))\n", "unmasked = in_batch_contrastive_loss(similarities, positive_indices=(0, 1))\n", "masked = in_batch_contrastive_loss(similarities, positive_indices=(0, 1), valid_mask=mask)\n", "print(\"Mask:\", mask)\n", "print(\"Unmasked loss:\", round(unmasked, 4))\n", "print(\"False-negative-aware loss:\", round(masked, 4))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Hard-negative mining needs an audit trail\n", "\n", "A robust loop retrieves with the current model, joins provenance/qrels,\n", "removes known and likely positives, samples across difficulty and source\n", "types, trains, and repeats on a frozen evaluation set. Include random/easy\n", "negatives so the model retains global separation; include adversarial\n", "lexical and semantic confounders; monitor how many mined “negatives” human\n", "adjudicators relabel as relevant.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Selected: ['hard-confounder', 'medium-confounder']\n", "Quarantined false negatives: ['same-source', 'answer-alias', 'teacher-says-positive']\n", "Easy/unselected: ['easy']\n" ] } ], "source": [ "from rag_evolution.training import NegativeExample, mine_hard_negatives\n", "\n", "pool = (\n", " NegativeExample(\"same-source\", 0.99, source_id=\"gold-source\"),\n", " NegativeExample(\"answer-alias\", 0.96, answer_ids=(\"rag\",)),\n", " NegativeExample(\"teacher-says-positive\", 0.91, teacher_relevance=0.8),\n", " NegativeExample(\"hard-confounder\", 0.88),\n", " NegativeExample(\"medium-confounder\", 0.63),\n", " NegativeExample(\"easy\", 0.05),\n", ")\n", "mining = mine_hard_negatives(\n", " pool, positive_source_ids=(\"gold-source\",), positive_answer_ids=(\"rag\",),\n", " k=2, minimum_score=0.5\n", ")\n", "print(\"Selected:\", [item.identifier for item in mining.selected])\n", "print(\"Quarantined false negatives:\", [item.identifier for item in mining.excluded_false_negatives])\n", "print(\"Easy/unselected:\", [item.identifier for item in mining.excluded_easy])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Retriever families learn different representations\n", "\n", "Dense bi-encoders learn one vector per query/passage (DPR, ANCE, RocketQA,\n", "Contriever, GTR, E5, DRAGON). Learned sparse models predict weighted\n", "vocabulary dimensions (DeepCT, DeepImpact, SPLADE). Late-interaction models\n", "retain token vectors and MaxSim interactions (ColBERT, PLAID, XTR, CITADEL).\n", "Reasoning-aware models train on “helpful versus plausible-but-unhelpful”\n", "documents. Unified models such as GritLM share embedding and generation.\n", "\n", "Pretraining choices—masked autoencoding, inverse cloze, synthetic queries,\n", "instruction data, domain adaptation, multilingual alignment—change transfer.\n", "Report model size, representation bytes, index size, query/document encoding\n", "cost, first-stage recall, and downstream utility.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Listwise weak/aligned: 1.2933 1.0133\n", "KL weak/aligned: 1.1492 0.0173\n" ] } ], "source": [ "from rag_evolution.training import kl_distillation_loss, listwise_cross_entropy\n", "\n", "teacher = (4.0, 2.0, 1.0, -1.0)\n", "weak_student = (1.0, 0.9, 0.8, 0.7)\n", "aligned_student = (3.8, 2.1, 1.0, -0.5)\n", "relevance = (3.0, 2.0, 1.0, 0.0)\n", "print(\"Listwise weak/aligned:\", round(listwise_cross_entropy(weak_student, relevance), 4), round(listwise_cross_entropy(aligned_student, relevance), 4))\n", "print(\"KL weak/aligned:\", round(kl_distillation_loss(weak_student, teacher, 2.0), 4), round(kl_distillation_loss(aligned_student, teacher, 2.0), 4))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Query transformation changes recall and can change intent\n", "\n", "Options include spelling/entity normalization, decomposition, multi-query\n", "paraphrases, pseudo-relevance feedback, HyDE hypothetical documents,\n", "Query2Doc expansion, step-back abstraction, conversation-history rewriting,\n", "metadata/temporal filters, and tool-selected structured queries. Transform\n", "quality must be judged against original intent; fluent rewrites can remove a\n", "constraint or invent a premise.\n", "\n", "Run each transform as an ablation and log the original query, every rewrite,\n", "retrieved set, new relevant evidence, duplicates, latency, and cost.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sparse: [('grip-2026', 1), ('rag-2020', 2), ('dpr-2020', 3), ('lara-2025', 4), ('search-r1-2025', 5)]\n", "Semantic proxy: [('rag-2020', 1), ('fid-2021', 2), ('crag-2024', 3), ('hyde-2022', 4), ('grip-2026', 5)]\n" ] } ], "source": [ "from rag_evolution.demo_data import demo_documents\n", "from rag_evolution.retrievers import BM25Retriever, HashingSemanticRetriever\n", "from rag_evolution.text import chunk_documents\n", "\n", "chunks = chunk_documents(demo_documents(), chunk_size=85, overlap=10)\n", "sparse = BM25Retriever(chunks)\n", "dense_proxy = HashingSemanticRetriever(chunks, dimensions=256)\n", "query = \"How do DPR and RAG differ in retrieval and generation?\"\n", "sparse_results = sparse.search(query, 8)\n", "dense_results = dense_proxy.search(query, 8)\n", "print(\"Sparse:\", [(item.chunk.document_id, item.rank) for item in sparse_results[:5]])\n", "print(\"Semantic proxy:\", [(item.chunk.document_id, item.rank) for item in dense_results[:5]])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Fusion: rank robustness versus score information\n", "\n", "Reciprocal-rank fusion (RRF) combines ordinal ranks and tolerates\n", "incomparable BM25/cosine scales. CombSUM/CombMNZ can exploit score magnitude\n", "only after calibration. Learned fusion can use query features and component\n", "scores but adds labels and shift risk. Missing candidates, depth, duplicate\n", "identities, and weights are part of the definition.\n", "\n", "A hybrid win does not reveal which component helped. Record per-result raw,\n", "calibrated, weighted, and fused scores and compare sparse-only, dense-only,\n", "union, RRF, calibrated score fusion, and reranked variants.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "RRF: [('rag-2020', 0.0635), ('grip-2026', 0.0608), ('crag-2024', 0.0581), ('dpr-2020', 0.0573), ('atlas-2022', 0.0548), ('fid-2021', 0.0312)]\n", "Calibrated CombSUM: [('rag-2020', 1.882), ('grip-2026', 1.294), ('fid-2021', 0.936), ('dpr-2020', 0.5), ('crag-2024', 0.346), ('lara-2025', 0.329)]\n", "Top CombSUM components: {'sparse_calibrated': 0.782026126654701, 'sparse_weighted': 0.782026126654701, 'sparse_raw': 6.002930239470717, 'semantic_calibrated': 1.0, 'semantic_weighted': 1.1, 'semantic_raw': 0.3039839749471921}\n" ] } ], "source": [ "from rag_evolution.selection import calibrated_comb_sum, reciprocal_rank_fusion\n", "\n", "rankings = {\"sparse\": sparse_results, \"semantic\": dense_results}\n", "rrf = reciprocal_rank_fusion(rankings, k=6, constant=30)\n", "comb = calibrated_comb_sum(rankings, k=6, weights={\"sparse\": 1.0, \"semantic\": 1.1})\n", "print(\"RRF:\", [(item.chunk.document_id, round(item.score, 4)) for item in rrf])\n", "print(\"Calibrated CombSUM:\", [(item.chunk.document_id, round(item.score, 3)) for item in comb])\n", "print(\"Top CombSUM components:\", dict(comb[0].component_scores))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Reranking crosses the query–document boundary\n", "\n", "Cross-encoders jointly attend to query and candidate and usually improve\n", "precision over independent embeddings. MonoT5/RankT5 cast ranking as\n", "generation; listwise LLM rerankers compare several candidates; late\n", "interaction lies between bi- and cross-encoders. Distill expensive teachers\n", "into cheaper rerankers, but validate calibration and position/order effects.\n", "\n", "First-stage recall remains a hard ceiling. Rerank enough candidates to expose\n", "relevant evidence, then report candidate recall, reranked nDCG/recall,\n", "latency, truncation, and cross-domain robustness.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Before: ['rag-2020', 'grip-2026', 'crag-2024', 'dpr-2020', 'atlas-2022', 'fid-2021']\n", "After: ['rag-2020', 'grip-2026', 'dpr-2020', 'crag-2024', 'atlas-2022', 'lara-2025']\n", "Interaction features: {'rerank_retrieval': 1.0, 'rerank_coverage': 0.6, 'rerank_phrase': 0.0, 'rerank_proximity': 0.333, 'rerank_title': 0.4, 'rerank_year': 1.0}\n" ] } ], "source": [ "from rag_evolution.rerankers import CrossFeatureReranker\n", "\n", "candidates = reciprocal_rank_fusion(rankings, k=10, constant=30)\n", "reranked = CrossFeatureReranker().rerank(query, candidates, k=6)\n", "print(\"Before:\", [item.chunk.document_id for item in candidates[:6]])\n", "print(\"After:\", [item.chunk.document_id for item in reranked])\n", "print(\"Interaction features:\", {k: round(v, 3) for k, v in reranked[0].component_scores.items() if k.startswith(\"rerank_\")})\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. The generator consumes a set, not a leaderboard\n", "\n", "Top-k can waste a budget on redundant passages while omitting a complementary\n", "fact. Evidence selection is a weighted set-cover/knapsack problem over\n", "claims, entities, sources, time versions, and token cost. Diversity/MMR is a\n", "useful proxy; explicit claim coverage is better when support annotations are\n", "available. Authority and conflict cannot be reduced to similarity alone.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Budget: 126 spent: 126\n", "Selected: ['rag-2020', 'dpr-2020']\n", "Covered/uncovered: ('generator', 'retriever', 'training') ('fusion',)\n" ] } ], "source": [ "from rag_evolution.selection import SelectionCandidate, greedy_budgeted_coverage\n", "from rag_evolution.text import tokenize\n", "\n", "supports = {\n", " \"dpr-2020\": (\"retriever\", \"training\"),\n", " \"rag-2020\": (\"retriever\", \"generator\"),\n", " \"fid-2020\": (\"generator\", \"fusion\"),\n", "}\n", "coverage_candidates = []\n", "for item in reranked:\n", " claims = supports.get(item.chunk.document_id, ())\n", " if claims:\n", " coverage_candidates.append(\n", " SelectionCandidate(item, claims, max(1, len(tokenize(item.chunk.text))))\n", " )\n", "budget = sum(sorted(candidate.cost for candidate in coverage_candidates)[:2])\n", "selection = greedy_budgeted_coverage(\n", " coverage_candidates,\n", " required=(\"retriever\", \"training\", \"generator\", \"fusion\"),\n", " budget=budget,\n", " relevance_weight=0.02,\n", ")\n", "print(\"Budget:\", budget, \"spent:\", selection.spent)\n", "print(\"Selected:\", [item.result.chunk.document_id for item in selection.selected])\n", "print(\"Covered/uncovered:\", selection.covered, selection.uncovered)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Preference and RL objectives need guarded rewards\n", "\n", "DPO can prefer cited, concise, abstaining, or low-cost trajectories relative\n", "to a reference policy. REINFORCE/GRPO/PPO-style optimization can learn\n", "retrieve/query/stop actions. Outcome-only answer rewards permit fabricated\n", "evidence, spurious search, or formatting hacks. Process rewards (support,\n", "information gain, redundancy, valid tool calls, calibrated stopping) help but\n", "are themselves gameable.\n", "\n", "Keep hard security/cost limits outside the learned policy. Audit reward\n", "correlation with human judgments, search traces, fabricated citations,\n", "over/under-search, transfer across corpora, and performance when retriever or\n", "generator changes.\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "DPO preferred/reversed: 0.513 0.913\n", "Returns: (0.865, 0.85, 1.0)\n", "Advantages: (0.665, 0.65, 0.8) loss 0.2282\n" ] } ], "source": [ "from rag_evolution.training import dpo_loss, reinforce_loss\n", "\n", "preferred = dpo_loss(-1.0, -3.0, -2.0, -2.0, beta=0.2)\n", "reversed_pair = dpo_loss(-3.0, -1.0, -2.0, -2.0, beta=0.2)\n", "trajectory = reinforce_loss(\n", " action_log_probabilities=(-0.3, -0.5, -0.2),\n", " rewards=(0.1, -0.05, 1.0),\n", " baseline=(0.2, 0.2, 0.2),\n", " discount=0.9,\n", ")\n", "print(\"DPO preferred/reversed:\", round(preferred, 4), round(reversed_pair, 4))\n", "print(\"Returns:\", tuple(round(value, 3) for value in trajectory.returns))\n", "print(\"Advantages:\", tuple(round(value, 3) for value in trajectory.advantages), \"loss\", round(trajectory.loss, 4))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 10. Attribute loss across the evidence pipeline\n", "\n", "Retrieval recall asks whether relevant evidence entered the candidate pool.\n", "Rerank survival asks whether it remained after second-stage selection. Pack\n", "survival asks whether it reached the model after deduplication and budgets.\n", "Context utilization asks whether the answer actually used it. Citation\n", "entailment/completeness ask whether claims point to supporting spans. One\n", "end-to-end score hides these failure locations.\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Recall/survival: {'retrieval': 1.0, 'rerank': 1.0, 'pack': 0.5, 'end_to_end': 0.5}\n", "Lost at stages: () () ('dpr-2020',)\n" ] } ], "source": [ "from rag_evolution.context import ContextPacker\n", "from rag_evolution.selection import evidence_flow\n", "\n", "packed = ContextPacker(max_tokens=180, max_chunks=3).pack(reranked)\n", "flow = evidence_flow((\"dpr-2020\", \"rag-2020\"), candidates, reranked, packed)\n", "print(\"Recall/survival:\", {\n", " \"retrieval\": round(flow.retrieval_recall, 3),\n", " \"rerank\": round(flow.rerank_survival, 3),\n", " \"pack\": round(flow.pack_survival, 3),\n", " \"end_to_end\": round(flow.end_to_end_recall, 3),\n", "})\n", "print(\"Lost at stages:\", flow.lost_at_retrieval, flow.lost_at_rerank, flow.lost_at_pack)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Experiment checklist\n", "\n", "Freeze corpus/qrels; record query and document encoders, prefixes, negatives,\n", "temperatures, mining checkpoint, fusion calibration, candidate depth,\n", "reranker truncation, pack budget, and seeds. Report per-query outputs and\n", "slices with paired confidence intervals. Evaluate BM25, dense, learned\n", "sparse, hybrid, reranked, oracle-context, and closed-book controls.\n", "\n", "**This lab does not reproduce** billion-parameter training or claim its\n", "hashed semantic proxy is neural retrieval. It makes objective functions and\n", "component boundaries executable so a real model can be substituted without\n", "changing the audit.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.9" }, "rag_evolution": { "presentation": "plain-field-notebook", "visual_version": 2 } }, "nbformat": 4, "nbformat_minor": 5 }