Teaching Search to Choose
The working page
This lab follows the learning signal through a modern retrieval stack. It treats negatives, score calibration, reranking, evidence-set selection, and retrieval-control rewards as first-class experimental variables.
Learning outcomes
- compute InfoNCE, pairwise, listwise, distillation, DPO, and policy-gradient objectives;
- identify false negatives and label leakage in hard-negative mining;
- compare score fusion with rank fusion;
- distinguish candidate ranking from budgeted evidence coverage;
- attribute relevant evidence lost at retrieval, reranking, or packing.
Companion chapters: Retrieval and ranking and Training and optimization.
from pathlib import Path
import sys
ROOT = Path.cwd()
if not (ROOT / "src").exists():
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / "src"))
print("Repository root: resolved from the notebook location")
Repository root: resolved from the notebook location
1. Labels define what “relevant” means
Positives may be human qrels, answer-containing passages, cited sources, supporting facts, clicked documents, successful tool results, synthetic teacher labels, or passages that improve a downstream reader. These signals disagree. Answer string containment can reward a passage that repeats a false claim; clicks encode position bias; citations may be incomplete; teacher labels inherit model bias; downstream utility can reward spurious shortcuts.
Preserve label provenance and uncertainty. Split by source/time/template to prevent leakage. Evaluate retriever recall, reader robustness, and generator parametric knowledge separately.
from rag_evolution.training import contrastive_loss, pairwise_hinge_loss, softmax
for temperature in (0.25, 0.5, 1.0, 2.0):
probabilities = softmax((3.0, 2.1, 0.5), temperature)
loss = contrastive_loss(3.0, (2.1, 0.5), temperature)
print("temperature", temperature, "P(positive)", round(probabilities[0], 4), "loss", round(loss, 4))
print("Hinge easy/hard:", pairwise_hinge_loss(3.0, 1.0), pairwise_hinge_loss(1.0, 2.5))
temperature 0.25 P(positive) 0.9734 loss 0.027 temperature 0.5 P(positive) 0.8532 loss 0.1587 temperature 1.0 P(positive) 0.6717 loss 0.3979 temperature 2.0 P(positive) 0.5197 loss 0.6545 Hinge easy/hard: 0.0 2.5
2. Contrastive learning is largely a negative-sampling design
For query (q_i), positive (d_i^+), negatives (d_j^-), a common loss is
In-batch negatives are cheap but may contain alternate positives. BM25/dense hard negatives teach fine distinctions but can concentrate annotation errors. Cross-encoder mining adds teacher bias; same-source negatives may be genuinely supportive. Track source identity, answer aliases, qrels, and teacher relevance, and quarantine candidates that may be false negatives.
from rag_evolution.training import false_negative_mask, in_batch_contrastive_loss
similarities = (
(3.2, 3.0, 0.2), # document 1 is an unlabeled alternate positive for query 0
(0.1, 3.1, 0.4),
)
mask = false_negative_mask(({"doc-0", "doc-1"}, {"doc-1"}), ("doc-0", "doc-1", "doc-2"))
unmasked = in_batch_contrastive_loss(similarities, positive_indices=(0, 1))
masked = in_batch_contrastive_loss(similarities, positive_indices=(0, 1), valid_mask=mask)
print("Mask:", mask)
print("Unmasked loss:", round(unmasked, 4))
print("False-negative-aware loss:", round(masked, 4))
Mask: ((False, False, True), (True, False, True)) Unmasked loss: 0.3679 False-negative-aware loss: 0.0796
3. Hard-negative mining needs an audit trail
A robust loop retrieves with the current model, joins provenance/qrels, removes known and likely positives, samples across difficulty and source types, trains, and repeats on a frozen evaluation set. Include random/easy negatives so the model retains global separation; include adversarial lexical and semantic confounders; monitor how many mined “negatives” human adjudicators relabel as relevant.
from rag_evolution.training import NegativeExample, mine_hard_negatives
pool = (
NegativeExample("same-source", 0.99, source_id="gold-source"),
NegativeExample("answer-alias", 0.96, answer_ids=("rag",)),
NegativeExample("teacher-says-positive", 0.91, teacher_relevance=0.8),
NegativeExample("hard-confounder", 0.88),
NegativeExample("medium-confounder", 0.63),
NegativeExample("easy", 0.05),
)
mining = mine_hard_negatives(
pool, positive_source_ids=("gold-source",), positive_answer_ids=("rag",),
k=2, minimum_score=0.5
)
print("Selected:", [item.identifier for item in mining.selected])
print("Quarantined false negatives:", [item.identifier for item in mining.excluded_false_negatives])
print("Easy/unselected:", [item.identifier for item in mining.excluded_easy])
Selected: ['hard-confounder', 'medium-confounder'] Quarantined false negatives: ['same-source', 'answer-alias', 'teacher-says-positive'] Easy/unselected: ['easy']
4. Retriever families learn different representations
Dense bi-encoders learn one vector per query/passage (DPR, ANCE, RocketQA, Contriever, GTR, E5, DRAGON). Learned sparse models predict weighted vocabulary dimensions (DeepCT, DeepImpact, SPLADE). Late-interaction models retain token vectors and MaxSim interactions (ColBERT, PLAID, XTR, CITADEL). Reasoning-aware models train on “helpful versus plausible-but-unhelpful” documents. Unified models such as GritLM share embedding and generation.
Pretraining choices—masked autoencoding, inverse cloze, synthetic queries, instruction data, domain adaptation, multilingual alignment—change transfer. Report model size, representation bytes, index size, query/document encoding cost, first-stage recall, and downstream utility.
from rag_evolution.training import kl_distillation_loss, listwise_cross_entropy
teacher = (4.0, 2.0, 1.0, -1.0)
weak_student = (1.0, 0.9, 0.8, 0.7)
aligned_student = (3.8, 2.1, 1.0, -0.5)
relevance = (3.0, 2.0, 1.0, 0.0)
print("Listwise weak/aligned:", round(listwise_cross_entropy(weak_student, relevance), 4), round(listwise_cross_entropy(aligned_student, relevance), 4))
print("KL weak/aligned:", round(kl_distillation_loss(weak_student, teacher, 2.0), 4), round(kl_distillation_loss(aligned_student, teacher, 2.0), 4))
Listwise weak/aligned: 1.2933 1.0133 KL weak/aligned: 1.1492 0.0173
5. Query transformation changes recall and can change intent
Options include spelling/entity normalization, decomposition, multi-query paraphrases, pseudo-relevance feedback, HyDE hypothetical documents, Query2Doc expansion, step-back abstraction, conversation-history rewriting, metadata/temporal filters, and tool-selected structured queries. Transform quality must be judged against original intent; fluent rewrites can remove a constraint or invent a premise.
Run each transform as an ablation and log the original query, every rewrite, retrieved set, new relevant evidence, duplicates, latency, and cost.
from rag_evolution.demo_data import demo_documents
from rag_evolution.retrievers import BM25Retriever, HashingSemanticRetriever
from rag_evolution.text import chunk_documents
chunks = chunk_documents(demo_documents(), chunk_size=85, overlap=10)
sparse = BM25Retriever(chunks)
dense_proxy = HashingSemanticRetriever(chunks, dimensions=256)
query = "How do DPR and RAG differ in retrieval and generation?"
sparse_results = sparse.search(query, 8)
dense_results = dense_proxy.search(query, 8)
print("Sparse:", [(item.chunk.document_id, item.rank) for item in sparse_results[:5]])
print("Semantic proxy:", [(item.chunk.document_id, item.rank) for item in dense_results[:5]])
Sparse: [('grip-2026', 1), ('rag-2020', 2), ('dpr-2020', 3), ('lara-2025', 4), ('search-r1-2025', 5)]
Semantic proxy: [('rag-2020', 1), ('fid-2021', 2), ('crag-2024', 3), ('hyde-2022', 4), ('grip-2026', 5)]
6. Fusion: rank robustness versus score information
Reciprocal-rank fusion (RRF) combines ordinal ranks and tolerates incomparable BM25/cosine scales. CombSUM/CombMNZ can exploit score magnitude only after calibration. Learned fusion can use query features and component scores but adds labels and shift risk. Missing candidates, depth, duplicate identities, and weights are part of the definition.
A hybrid win does not reveal which component helped. Record per-result raw, calibrated, weighted, and fused scores and compare sparse-only, dense-only, union, RRF, calibrated score fusion, and reranked variants.
from rag_evolution.selection import calibrated_comb_sum, reciprocal_rank_fusion
rankings = {"sparse": sparse_results, "semantic": dense_results}
rrf = reciprocal_rank_fusion(rankings, k=6, constant=30)
comb = calibrated_comb_sum(rankings, k=6, weights={"sparse": 1.0, "semantic": 1.1})
print("RRF:", [(item.chunk.document_id, round(item.score, 4)) for item in rrf])
print("Calibrated CombSUM:", [(item.chunk.document_id, round(item.score, 3)) for item in comb])
print("Top CombSUM components:", dict(comb[0].component_scores))
RRF: [('rag-2020', 0.0635), ('grip-2026', 0.0608), ('crag-2024', 0.0581), ('dpr-2020', 0.0573), ('atlas-2022', 0.0548), ('fid-2021', 0.0312)]
Calibrated CombSUM: [('rag-2020', 1.882), ('grip-2026', 1.294), ('fid-2021', 0.936), ('dpr-2020', 0.5), ('crag-2024', 0.346), ('lara-2025', 0.329)]
Top CombSUM components: {'sparse_calibrated': 0.782026126654701, 'sparse_weighted': 0.782026126654701, 'sparse_raw': 6.002930239470717, 'semantic_calibrated': 1.0, 'semantic_weighted': 1.1, 'semantic_raw': 0.3039839749471921}
7. Reranking crosses the query–document boundary
Cross-encoders jointly attend to query and candidate and usually improve precision over independent embeddings. MonoT5/RankT5 cast ranking as generation; listwise LLM rerankers compare several candidates; late interaction lies between bi- and cross-encoders. Distill expensive teachers into cheaper rerankers, but validate calibration and position/order effects.
First-stage recall remains a hard ceiling. Rerank enough candidates to expose relevant evidence, then report candidate recall, reranked nDCG/recall, latency, truncation, and cross-domain robustness.
from rag_evolution.rerankers import CrossFeatureReranker
candidates = reciprocal_rank_fusion(rankings, k=10, constant=30)
reranked = CrossFeatureReranker().rerank(query, candidates, k=6)
print("Before:", [item.chunk.document_id for item in candidates[:6]])
print("After:", [item.chunk.document_id for item in reranked])
print("Interaction features:", {k: round(v, 3) for k, v in reranked[0].component_scores.items() if k.startswith("rerank_")})
Before: ['rag-2020', 'grip-2026', 'crag-2024', 'dpr-2020', 'atlas-2022', 'fid-2021']
After: ['rag-2020', 'grip-2026', 'dpr-2020', 'crag-2024', 'atlas-2022', 'lara-2025']
Interaction features: {'rerank_retrieval': 1.0, 'rerank_coverage': 0.6, 'rerank_phrase': 0.0, 'rerank_proximity': 0.333, 'rerank_title': 0.4, 'rerank_year': 1.0}
8. The generator consumes a set, not a leaderboard
Top-k can waste a budget on redundant passages while omitting a complementary fact. Evidence selection is a weighted set-cover/knapsack problem over claims, entities, sources, time versions, and token cost. Diversity/MMR is a useful proxy; explicit claim coverage is better when support annotations are available. Authority and conflict cannot be reduced to similarity alone.
from rag_evolution.selection import SelectionCandidate, greedy_budgeted_coverage
from rag_evolution.text import tokenize
supports = {
"dpr-2020": ("retriever", "training"),
"rag-2020": ("retriever", "generator"),
"fid-2020": ("generator", "fusion"),
}
coverage_candidates = []
for item in reranked:
claims = supports.get(item.chunk.document_id, ())
if claims:
coverage_candidates.append(
SelectionCandidate(item, claims, max(1, len(tokenize(item.chunk.text))))
)
budget = sum(sorted(candidate.cost for candidate in coverage_candidates)[:2])
selection = greedy_budgeted_coverage(
coverage_candidates,
required=("retriever", "training", "generator", "fusion"),
budget=budget,
relevance_weight=0.02,
)
print("Budget:", budget, "spent:", selection.spent)
print("Selected:", [item.result.chunk.document_id for item in selection.selected])
print("Covered/uncovered:", selection.covered, selection.uncovered)
Budget: 126 spent: 126
Selected: ['rag-2020', 'dpr-2020']
Covered/uncovered: ('generator', 'retriever', 'training') ('fusion',)
9. Preference and RL objectives need guarded rewards
DPO can prefer cited, concise, abstaining, or low-cost trajectories relative to a reference policy. REINFORCE/GRPO/PPO-style optimization can learn retrieve/query/stop actions. Outcome-only answer rewards permit fabricated evidence, spurious search, or formatting hacks. Process rewards (support, information gain, redundancy, valid tool calls, calibrated stopping) help but are themselves gameable.
Keep hard security/cost limits outside the learned policy. Audit reward correlation with human judgments, search traces, fabricated citations, over/under-search, transfer across corpora, and performance when retriever or generator changes.
from rag_evolution.training import dpo_loss, reinforce_loss
preferred = dpo_loss(-1.0, -3.0, -2.0, -2.0, beta=0.2)
reversed_pair = dpo_loss(-3.0, -1.0, -2.0, -2.0, beta=0.2)
trajectory = reinforce_loss(
action_log_probabilities=(-0.3, -0.5, -0.2),
rewards=(0.1, -0.05, 1.0),
baseline=(0.2, 0.2, 0.2),
discount=0.9,
)
print("DPO preferred/reversed:", round(preferred, 4), round(reversed_pair, 4))
print("Returns:", tuple(round(value, 3) for value in trajectory.returns))
print("Advantages:", tuple(round(value, 3) for value in trajectory.advantages), "loss", round(trajectory.loss, 4))
DPO preferred/reversed: 0.513 0.913 Returns: (0.865, 0.85, 1.0) Advantages: (0.665, 0.65, 0.8) loss 0.2282
10. Attribute loss across the evidence pipeline
Retrieval recall asks whether relevant evidence entered the candidate pool. Rerank survival asks whether it remained after second-stage selection. Pack survival asks whether it reached the model after deduplication and budgets. Context utilization asks whether the answer actually used it. Citation entailment/completeness ask whether claims point to supporting spans. One end-to-end score hides these failure locations.
from rag_evolution.context import ContextPacker
from rag_evolution.selection import evidence_flow
packed = ContextPacker(max_tokens=180, max_chunks=3).pack(reranked)
flow = evidence_flow(("dpr-2020", "rag-2020"), candidates, reranked, packed)
print("Recall/survival:", {
"retrieval": round(flow.retrieval_recall, 3),
"rerank": round(flow.rerank_survival, 3),
"pack": round(flow.pack_survival, 3),
"end_to_end": round(flow.end_to_end_recall, 3),
})
print("Lost at stages:", flow.lost_at_retrieval, flow.lost_at_rerank, flow.lost_at_pack)
Recall/survival: {'retrieval': 1.0, 'rerank': 1.0, 'pack': 0.5, 'end_to_end': 0.5}
Lost at stages: () () ('dpr-2020',)
Experiment checklist
Freeze corpus/qrels; record query and document encoders, prefixes, negatives, temperatures, mining checkpoint, fusion calibration, candidate depth, reranker truncation, pack budget, and seeds. Report per-query outputs and slices with paired confidence intervals. Evaluate BM25, dense, learned sparse, hybrid, reranked, oracle-context, and closed-book controls.
This lab does not reproduce billion-parameter training or claim its hashed semantic proxy is neural retrieval. It makes objective functions and component boundaries executable so a real model can be substituted without changing the audit.