Training and optimizing every RAG component

RAG can be assembled from frozen components, but frontier systems learn one or more of: representation, ranking, evidence selection, retrieval timing, query generation, stopping, grounding, citations, or memory policy. The central problem is credit assignment: answer correctness does not reveal whether the retriever found the right evidence or whether the generator used it.

1. Define the training target before the loss

A training example may contain

(q, D^+, D^-, y, C, A, \tau, m),

where $q$ is a query, $D^+$ supporting evidence, $D^-$ non-supporting candidates, $y$ an answer, $C$ atomic claims, $A$ claim-evidence links, $\tau$ a search trajectory, and $m$ metadata such as time, permissions, language, and source authority. Most public datasets provide only a subset.

Different targets require different labels:

topical relevance;
answer-containing passage;
sufficient supporting evidence;
complete evidence set;
downstream answer utility;
citation entailment and authority;
retrieve/no-retrieve decision;
next search query/action;
stop decision;
calibrated abstention;
latency/cost/risk under a budget.

Do not train “relevance” on answer-containing heuristics and later interpret the score as causal support or source authority.

2. Data sources and their biases

2.1 Human qrels and support annotations

Human judgments are the strongest target when guidelines distinguish relevant, supporting, complete, contradictory, and authoritative evidence. They are expensive and incomplete. Pool documents from diverse retrievers for judging, double-label a sample, adjudicate, report agreement, and retain uncertainty.

2.2 QA evidence and citations

Wikipedia QA, fact checking, scientific citations, and attributed answers provide evidence links. A cited page may be too coarse, incomplete, or chosen for presentation rather than minimal support. Map pages to exact spans where possible and preserve alternative evidence sets.

2.3 Answer-string distant supervision

Treat a passage containing an answer string as positive. This enabled large open-QA datasets for ORQA/DPR-era systems but admits spurious mentions, wrong relations, wrong dates, and copied answer lists. Questions with no answer string are often dropped, biasing training toward retrievable/extractive cases.

2.4 Behavioral feedback

Clicks, dwell, copied citations, accepted answers, ticket resolution, user edits, and follow-up queries are implicit signals. They suffer position, UI, popularity, selection, and satisfaction bias. Log exposure propensities and use causal/ counterfactual methods when feasible. Never interpret an unclicked result as a clean negative when the user may not have seen it.

2.5 Synthetic queries and labels

Generate questions from documents, summaries, propositions, or tables; use an LLM/cross-encoder as teacher; create counterfactuals, conflicts, and no-answer examples. Synthetic data scales and covers domain terminology, but inherits teacher style, blind spots, and leakage. Filter for answerability, diversity, source alignment, and near-duplicates; evaluate on independent human data.

2.6 Self-training and mined trajectories

Run a system, keep high-reward searches/answers, and train on them. This can improve a policy but narrows exploration and amplifies evaluator bias. Preserve failed/alternative trajectories and periodically refresh from human judgments.

3. Positive construction

Positives can be document, passage, sentence, proposition, table row, graph path, page, image region, or set. Match training granularity to serving granularity. If only document labels exist but passages are indexed, select or soft-label passages rather than declaring every passage positive.

Use multiple positives when several sources support the answer. A supervised contrastive objective can sum over positives:

\mathcal L_i=-\log \frac{\sum_{d\in P_i}\exp s(q_i,d)/\tau} {\sum_{d\in P_i\cup N_i}\exp s(q_i,d)/\tau}.

For multi-hop questions, label both individual supporting units and complete sets/paths. Training only individual relevance does not teach evidence composition.

4. Negative sampling taxonomy

4.1 Random negatives

Cheap and useful at the beginning, but usually topically trivial. The model learns broad domain separation rather than fine relevance.

4.2 In-batch negatives

Other examples’ positives become negatives. For batch $B$ , each query sees $|B|-1$ negatives with no additional encoding. Large and diverse batches improve signal, but duplicate topics and alternative valid evidence create false negatives. Cross-device gathering enlarges the pool and communication cost.

4.3 Cross-batch memory

Queue earlier document embeddings, as in contrastive vision/language training. This creates more negatives but embeddings may be stale relative to current parameters. Track queue age and avoid treating known positives as negatives.

4.4 BM25 hard negatives

Select high lexical matches that lack a positive label. DPR showed their value: the model must distinguish same terms with wrong semantics. They are vulnerable to incomplete qrels and answer-string false negatives.

4.5 ANN-mined negatives

ANCE periodically embeds the corpus with a current or recent encoder, builds ANN, and retrieves each query’s nearest nonpositive documents. This targets the model’s active confusions.

Index refresh is asynchronous and expensive. If too stale, negatives no longer match the model; if refreshed constantly, training stalls on re-encoding. Record model/index generation and mining cadence.

4.6 Teacher-denoised negatives

RocketQA uses a cross-encoder to filter noisy hard negatives and augment positives. A teacher can distinguish same-answer or actually relevant candidates, but transfers its biases. Audit teacher errors, especially on long, multilingual, temporal, and table evidence.

4.7 Adversarial and counterfactual negatives

Construct same-entity/wrong-relation, wrong-date/version, negated, unit-swapped, source-spoofed, and nearly supporting passages. These train the distinctions RAG needs. Generated counterfactuals must not accidentally remain true or become unnatural shortcuts.

4.8 False-negative mitigation

retrieve/judge alternative positives;
mask duplicate or same-answer candidates;
use soft teacher relevance instead of binary labels;
use debiased contrastive losses;
downweight ambiguous candidates;
inspect hard-negative clusters manually;
retain source/time/scope metadata in labels;
evaluate against pooled, expanded qrels.

Harder is not always better: a false negative near the decision boundary creates a large harmful gradient.

5. Dense retriever objectives

5.1 Multiple-negative softmax / InfoNCE

\mathcal L_i=-\log \frac{e^{s(q_i,d_i^+)/\tau}} {e^{s(q_i,d_i^+)/\tau}+\sum_j e^{s(q_i,d_{ij}^-)/\tau}}.

Temperature controls concentration. With dot product, model norms can change effective temperature; L2 normalization removes that degree of freedom. Batch composition and number of negatives materially change the loss.

5.2 Triplet and margin loss

\mathcal L_i=\max(0,m-s(q_i,d_i^+)+s(q_i,d_i^-)).

It focuses on violations but ignores already separated pairs and requires a margin. Multiple hard negatives or smooth softplus variants improve training.

5.3 Pairwise logistic loss

\mathcal L_i=\log(1+e^{-(s^+-s^-)}).

This gives continuous gradients and is common in ranking/distillation.

5.4 Listwise likelihood

Normalize over a candidate list and match one-hot or graded relevance. Listwise training better reflects ordering, but candidates and incomplete labels define the target distribution.

5.5 Margin-MSE distillation

GPL matches teacher score margins:

\mathcal L=(s_\theta(q,d^+)-s_\theta(q,d^-) -[s_T(q,d^+)-s_T(q,d^-)])^2.

Margins carry graded preference without requiring comparable absolute teacher and student scales. Teacher calibration and candidate sampling still matter.

5.6 KL/listwise distillation

For teacher distribution $p_T(d\mid q)$ and student $p_S$ , minimize

\operatorname{KL}(p_T\|p_S)=\sum_d p_T(d\mid q) \log\frac{p_T(d\mid q)}{p_S(d\mid q)}.

Temperature can expose dark knowledge among negatives. Candidate lists must include meaningful alternatives; distillation cannot teach unseen distinctions.

6. Retrieval-oriented pretraining

Inverse cloze task

Select a sentence as pseudo-query and its surrounding context as positive. ORQA used ICT to overcome latent-retrieval cold start. It supplies massive weak data but may teach document-local rather than user-query relevance.

Condenser

Condenser uses an architecture that forces late MLM processing through a [CLS] representation, making the bottleneck useful before retrieval fine-tuning.

coCondenser

coCondenser adds corpus-aware contrastive learning between spans of the same document. It creates global semantic structure without labeled queries.

RetroMAE

RetroMAE gives a full encoder a lightly masked input and a shallow decoder a heavily masked view, forcing the sentence embedding to preserve reconstructive information.

SimLM

SimLM uses a bottlenecked replaced-language-modeling objective. Its goal is to make the single vector carry token-level information relevant to retrieval.

Weakly supervised pair pretraining

E5-style training collects many text-pair relations and uses large-batch contrastive learning with explicit query/passage prefixes. INSTRUCTOR conditions embeddings on task instructions. Generality depends on pair diversity, instruction coverage, and contamination.

7. Learned sparse objectives

SPLADE-style systems optimize ranking loss plus query/document sparsity:

\mathcal L=\mathcal L_{rank} +\lambda_q\mathcal R(w(q)) +\lambda_d\mathcal R(w(d)).

FLOPS regularization penalizes vocabulary dimensions frequently active over a batch, approximating posting-list work. L1 penalizes each representation’s mass. Query and document costs differ: query nonzeros affect lists opened; document nonzeros affect index size/posting density.

Training choices include MLM initialization, hard negatives, cross-encoder distillation, ensemble teachers, self-distillation, quantization, and curriculum on regularization. Report effectiveness at matched index/latency budgets, not only the best unconstrained model.

8. Multi-vector training

ColBERT optimizes passage ranking while retaining token vectors. MaxSim creates hard discrete token alignments; in-batch negatives teach which local matches matter. ColBERTv2 uses denoised supervision and residual compression. Later systems train routing or token retrievability to reduce scoring cost.

Negative construction should include passages with overlapping keywords but wrong relations, because MaxSim can overvalue isolated term matches. Training must match the serving compression/index; quantization-aware or centroid-aware distillation may reduce train/serve mismatch.

Visual late-interaction models train query-token/page-patch alignment from document retrieval pairs, often with synthetic or weak labels. Evaluate domain overlap, language, page-level ambiguity, layout degradation, and storage.

9. Reranker training

Pointwise

Train binary/graded relevance per query-document pair. Easy to calibrate and batch, but independent scores ignore list/set structure.

Pairwise

Optimize positive over negative. Sample pairs across rank positions and retriever sources. If training negatives come only from BM25, the model may not learn dense-retriever errors.

Listwise and Lambda losses

Optimize a list distribution or weight pairwise gradients by metric change, such as $|\Delta\mathrm{nDCG}|$ . This aligns ranking metrics but qrel incompleteness can heavily penalize unjudged valid evidence.

Generative ranking

monoT5 generates a relevance token. RankT5 directly predicts scores. LLM rerankers output permutations or pairwise preferences. Distill expensive teachers into smaller cross-encoders, retaining hard candidate lists and teacher uncertainty.

Downstream-utility training

Relevance is not identical to usefulness. Train a selector using generator loss, answer correctness, claim coverage, or marginal contribution. Control for generator parametric knowledge: a passage may appear useless because the model already knows the answer, or useful only to one generator.

10. Query rewriter and decomposer training

Supervision can be human standalone rewrites, teacher rewrites, clicked/relevant documents, or downstream answer reward. Text-likelihood training copies plausible rewrites but may not optimize retrieval. A retrieval-aware objective maximizes relevant-document probability:

\mathcal L_{rewrite}=-\log \sum_{d\in D^+}p_\eta(d\mid \hat q_\phi(q,h)).

Jointly training through a discrete search/query string is difficult; use policy gradient, sequence-level distillation, or differentiable retriever proxies.

For decomposition, label subquestions, dependencies, required evidence, and stop state. Penalize redundant/unanswerable subquestions and fan-out cost. Always test intent preservation and direct-query fallback.

11. Generator supervised fine-tuning

Train on (instruction, evidence, answer, citations) examples with varied evidence conditions:

complete gold support;
partial support;
no support;
irrelevant distractors;
same-entity hard distractors;
conflicts and stale/current versions;
source-authority differences;
malicious embedded instructions;
long and reordered contexts.

If training always supplies perfect context, the model learns neither robust selection nor abstention. If citations always appear at answer end, it will not learn local attribution.

Token likelihood

\mathcal L_{gen}=-\sum_t\log p_\theta(y_t\mid y_{<t},q,Z).

This rewards copying reference style and content. It does not separately reward factuality, completeness, support, or citation validity.

Evidence dropout and distractor curricula

Randomly remove supporting units, add negatives, change order, or vary evidence budget. Teach the output state (answer, partial, insufficient, conflict) and ensure removal does not create incorrectly labeled examples.

Citation training

Use stable evidence IDs and attach them to atomic claims. Loss can include citation tokens plus auxiliary support/alignment objectives. Negative examples include wrong-but-retrieved IDs, non-supporting mentions, and fabricated IDs. Validate citation existence deterministically at inference.

12. Latent-document joint training

RAG/REALM-style models optimize

\mathcal L=-\log\sum_{z\in\operatorname{TopK}_\eta(q)} p_\eta(z\mid q)p_\theta(y\mid q,z).

The generator likelihood supplies a retriever signal: documents making the answer likely gain probability. Problems:

top-k truncation gives zero gradient to unretrieved evidence;
generator may assign high likelihood to spurious answer-containing passages;
parametric knowledge weakens evidence credit;
document embeddings/index become stale;
re-indexing is expensive and nondifferentiable;
likelihood does not enforce citation or authority.

Warm-start with retrieval supervision/pretraining, refresh indexes, and combine explicit evidence labels or distillation.

13. EM and multi-stage joint learning

Treat evidence as latent and alternate:

E-like step: estimate a posterior or reader utility over documents;
M-like step: train retriever and generator from that distribution.

EMDR² and related methods approximate end-to-end learning for multi-document QA. Teacher posteriors can be sharp and unstable; evidence sets interact; the candidate pool remains a ceiling. Log posterior entropy and whether mass falls on genuinely supporting documents.

14. Reader-to-retriever distillation

FiD-KD transfers reader signals—attention or passage contribution—to a retriever. Let teacher logits $t_i$ and retriever logits $s_i$ :

\mathcal L_{KD}=\operatorname{KL} (\operatorname{softmax}(t/T)\|\operatorname{softmax}(s/T)).

Attention is not guaranteed causal evidence use. Alternative leave-one-out utility measures the answer-likelihood change after removing a passage, at greater cost. Distillation is tied to reader, prompt, and candidate set.

15. Preference optimization for grounded generation

Construct preferred/rejected answers or trajectories differing in correctness, support, completeness, citation, abstention, and cost. Direct Preference Optimization uses

\mathcal L_{DPO}=-\log\sigma\left( \beta\log\frac{\pi_\theta(y^+\mid x)}{\pi_{ref}(y^+\mid x)} -\beta\log\frac{\pi_\theta(y^-\mid x)}{\pi_{ref}(y^-\mid x)} \right).

Pairs should isolate the desired property. If preferred answers are also longer or more stylistically polished, the model learns that shortcut. Include faithful concise and unfaithful fluent contrasts, and audit by claim.

16. Reinforcement learning for search

Model retrieval control as an MDP:

state $s_t$ : request, reasoning state, selected evidence, remaining budget;
action $a_t$ : no-search, query, retriever/source, select, read, answer, stop;
transition: search/tool result and model state update;
reward: correctness, support, citation, process quality, latency/cost/risk;
terminal condition: answer, abstain, or hard budget.

Policy gradient

\nabla J(\theta)= \mathbb E_\tau\left[ \sum_t\nabla\log\pi_\theta(a_t\mid s_t)(R(\tau)-b_t) \right].

PPO clips policy-ratio changes; GRPO-style methods use group-relative advantages without a learned critic; DAPO and variants modify sampling and optimization. The algorithm name is less important than action validity, reward definition, exploration, and corpus/retriever setup.

Outcome rewards

Exact match/F1 or an answer judge is easy but cannot prove good search. The model may answer from memory, fabricate retrieval tags, exploit formatting, or use irrelevant passages. ReSearch and Search-R1 demonstrate strong outcome-trained search while retaining these attribution caveats.

Process rewards

Reward useful search timing, information gain, nonredundancy, evidence support, and correct stop. StepSearch uses stepwise process signals; HiPRAG defines hierarchical rewards based on optimal search decisions. Process labels can be synthetic or task-specific and may not transfer.

Value-based evidence selection

Q-RAG freezes the LLM and learns an embedding-based Q-function over candidate selection/STOP actions. It separates retriever-policy learning from generator weights and can scale selection, but relies on support-fact supervision and its defined terminal reward.

Cost and risk rewards

Use explicit constraints or Lagrangian penalties:

R=R_{quality}+\alpha R_{support} -\lambda_1N_{search}-\lambda_2\text{tokens} -\lambda_3\text{latency}-\lambda_4\text{risk}.

Average penalties do not guarantee per-request limits. Enforce maximum calls, source permissions, and timeouts outside the policy.

17. Reward design and hacking tests

Separate reward components and log each:

answer correctness;
completeness;
citation entailment and completeness;
evidence-set recall/authority/freshness;
calibrated abstention;
valid query/action syntax;
unique useful evidence and information gain;
step/call/token/latency/cost;
safety and policy compliance.

Red-team the reward:

fabricated <search>/document tags;
answering without searching while claiming retrieval;
copying reference phrases;
over-search to accumulate process points;
trivial stop to avoid cost;
citation to any retrieved document regardless of support;
judge-prompt injection from evidence;
length/style shortcuts;
exploiting known benchmark answer formats.

Use held-out judges, human audits, counterfactual evidence swaps, and tests where parametric knowledge conflicts with documents.

18. Curriculum and staged training

A practical sequence is:

retrieval-oriented encoder pretraining or strong pretrained embeddings;
supervised/weak retriever training with in-batch and lexical negatives;
ANN mining and teacher denoising;
reranker training on the deployed hybrid candidate distribution;
generator SFT with complete/partial/noisy/conflicting evidence and citations;
reader-to-retriever or utility distillation;
supervised search/control trajectories;
preference/RL optimization under hard budgets;
domain calibration and human evaluation;
continual mining/retraining with drift and rollback controls.

Jointly optimizing everything from random initialization is rarely stable or necessary. Keep strong modular baselines to locate gains.

19. Domain adaptation

Retriever

continue MLM/retrieval pretraining on unlabeled domain text;
generate domain queries and teacher margins (GPL-style);
collect human queries/qrels;
mine hard negatives from the target corpus;
preserve general-domain mixtures to reduce catastrophic forgetting;
tune analyzers and retain BM25/hybrid baselines.

Generator

teach domain terminology and output schemas;
include evidence authority/time rules and abstention;
use retrieved context conditions matching deployment;
keep evaluation separate from synthetic teacher generation.

Specialized domains

Medical/legal/financial models require authoritative-source and temporal labels, not merely domain vocabulary. Code retrievers need repository structure and revision-aware negatives. Multilingual adaptation needs language-balanced batches and cross-lingual positives; English teacher quality can create unequal performance.

20. Continual learning and index consistency

Corpus change creates new positives, negatives, terms, entities, and embedding distribution. Monitor:

retrieval score/rank distributions;
new-query and no-result rate;
language/domain/tenant mix;
qrel and answer quality by time;
hard-negative composition;
embedding norms and ANN recall;
stale-answer and superseded-source rate.

Retraining changes the vector space. Build a new index generation, evaluate paired traffic, and cut over atomically. Online query-encoder updates against old document embeddings are unsafe unless compatibility is explicitly trained and verified.

Avoid feedback loops where only retrieved/shown/clicked documents become future positives. Maintain exploration, diverse judging pools, and counterfactual evaluation.

21. Training security and privacy

remove secrets/PII or use approved local processing;
preserve consent/license/training-eligibility metadata;
defend against poisoned documents and synthetic labels;
isolate untrusted text from teacher/system prompts;
track source lineage into training examples;
support deletion/unlearning policy across datasets/checkpoints where promised;
audit memorization and corpus extraction;
prevent cross-tenant examples and batch negatives;
secure model, index, and trajectory artifacts.

RAG’s external datastore can reduce the need to bake private facts into weights, but training on retrieved or logged private content recreates the problem.

22. Evaluation protocol for training claims

Frozen component evaluations

retriever with fixed corpus/chunker/index;
reranker on a frozen candidate pool;
generator with gold and controlled contexts;
policy with a frozen retriever/generator/tool environment.

End-to-end evaluation

After component gains, rebuild the complete pipeline and measure answer, citations, abstention, latency, cost, and risk. A retriever nDCG gain can add longer/noisier contexts and reduce generation quality.

Statistical design

paired per-query comparisons;
confidence intervals/bootstrap or randomization tests;
multiple seeds for stochastic training and RL;
versioned model/index/corpus/prompts;
contamination checks;
slice metrics and worst-group behavior;
human audit of high-impact disagreements;
compute/energy and training/inference resource report.

Ablations

Remove pretraining, negative types, teacher denoising, distillation, process rewards, cost penalty, citations, and curriculum stages individually. Report the candidate/index setup for each. If several changes enter together, the source of improvement is unknown.

23. Choosing what to train

Constraint/problem	First training target	Why
No labels, broad domain	strong general embedding + BM25; synthetic/GPL adaptation	avoids premature custom model
Vocabulary/domain mismatch	retriever/analyzer with target qrels	directly improves recall ceiling
High first-stage recall, poor precision	reranker/selector on deployed candidates	cheaper than retraining corpus embeddings
Evidence present but ignored	generator SFT with noisy/partial contexts	retrieval is not the bottleneck
Missing multi-hop evidence	decomposer/search policy + set supervision	one-shot ranker objective is wrong
Over-retrieval/cost	retrieve/stop policy with quality-cost objective	adapt budget by query
Citation errors	claim/evidence SFT and verifier	answer correctness alone insufficient
Domain shift	domain queries, hard negatives, calibration	benchmark-general model may not transfer
Frequent corpus change	modular retriever/index + continual evaluation	joint retraining may be too slow
High privacy/regulation	external governed memory; minimal private fine-tuning	improves deletion/audit boundary

24. What the executable notebooks model

The training notebook computes multiple-negative, margin, pairwise, listwise, distillation, DPO, and policy-gradient objectives on inspectable examples. It demonstrates random versus in-batch versus lexical/ANN hard negatives, false negative masking, teacher soft labels, reward decomposition, and cost-constrained search trajectories. It also shows why a correct answer can assign the wrong retriever credit.

No notebook trains a billion-parameter model. The objective is to make the mathematics, label assumptions, and failure modes executable before substituting real encoders, LLMs, distributed indexes, or RL infrastructure.