# Training and optimizing every RAG component

RAG can be assembled from frozen components, but frontier systems learn one or
more of: representation, ranking, evidence selection, retrieval timing, query
generation, stopping, grounding, citations, or memory policy. The central
problem is credit assignment: answer correctness does not reveal whether the
retriever found the right evidence or whether the generator used it.

## 1. Define the training target before the loss

A training example may contain

\[
(q, D^+, D^-, y, C, A, \tau, m),
\]

where \(q\) is a query, \(D^+\) supporting evidence, \(D^-\) non-supporting
candidates, \(y\) an answer, \(C\) atomic claims, \(A\) claim-evidence links,
\(\tau\) a search trajectory, and \(m\) metadata such as time, permissions,
language, and source authority. Most public datasets provide only a subset.

Different targets require different labels:

- topical relevance;
- answer-containing passage;
- sufficient supporting evidence;
- complete evidence set;
- downstream answer utility;
- citation entailment and authority;
- retrieve/no-retrieve decision;
- next search query/action;
- stop decision;
- calibrated abstention;
- latency/cost/risk under a budget.

Do not train “relevance” on answer-containing heuristics and later interpret the
score as causal support or source authority.

## 2. Data sources and their biases

### 2.1 Human qrels and support annotations

Human judgments are the strongest target when guidelines distinguish relevant,
supporting, complete, contradictory, and authoritative evidence. They are
expensive and incomplete. Pool documents from diverse retrievers for judging,
double-label a sample, adjudicate, report agreement, and retain uncertainty.

### 2.2 QA evidence and citations

Wikipedia QA, fact checking, scientific citations, and attributed answers
provide evidence links. A cited page may be too coarse, incomplete, or chosen
for presentation rather than minimal support. Map pages to exact spans where
possible and preserve alternative evidence sets.

### 2.3 Answer-string distant supervision

Treat a passage containing an answer string as positive. This enabled large
open-QA datasets for ORQA/DPR-era systems but admits spurious mentions, wrong
relations, wrong dates, and copied answer lists. Questions with no answer string
are often dropped, biasing training toward retrievable/extractive cases.

### 2.4 Behavioral feedback

Clicks, dwell, copied citations, accepted answers, ticket resolution, user edits,
and follow-up queries are implicit signals. They suffer position, UI, popularity,
selection, and satisfaction bias. Log exposure propensities and use causal/
counterfactual methods when feasible. Never interpret an unclicked result as a
clean negative when the user may not have seen it.

### 2.5 Synthetic queries and labels

Generate questions from documents, summaries, propositions, or tables; use an
LLM/cross-encoder as teacher; create counterfactuals, conflicts, and no-answer
examples. Synthetic data scales and covers domain terminology, but inherits
teacher style, blind spots, and leakage. Filter for answerability, diversity,
source alignment, and near-duplicates; evaluate on independent human data.

### 2.6 Self-training and mined trajectories

Run a system, keep high-reward searches/answers, and train on them. This can
improve a policy but narrows exploration and amplifies evaluator bias. Preserve
failed/alternative trajectories and periodically refresh from human judgments.

## 3. Positive construction

Positives can be document, passage, sentence, proposition, table row, graph
path, page, image region, or set. Match training granularity to serving
granularity. If only document labels exist but passages are indexed, select or
soft-label passages rather than declaring every passage positive.

Use multiple positives when several sources support the answer. A supervised
contrastive objective can sum over positives:

\[
\mathcal L_i=-\log
\frac{\sum_{d\in P_i}\exp s(q_i,d)/\tau}
{\sum_{d\in P_i\cup N_i}\exp s(q_i,d)/\tau}.
\]

For multi-hop questions, label both individual supporting units and complete
sets/paths. Training only individual relevance does not teach evidence
composition.

## 4. Negative sampling taxonomy

### 4.1 Random negatives

Cheap and useful at the beginning, but usually topically trivial. The model
learns broad domain separation rather than fine relevance.

### 4.2 In-batch negatives

Other examples’ positives become negatives. For batch \(B\), each query sees
\(|B|-1\) negatives with no additional encoding. Large and diverse batches
improve signal, but duplicate topics and alternative valid evidence create false
negatives. Cross-device gathering enlarges the pool and communication cost.

### 4.3 Cross-batch memory

Queue earlier document embeddings, as in contrastive vision/language training.
This creates more negatives but embeddings may be stale relative to current
parameters. Track queue age and avoid treating known positives as negatives.

### 4.4 BM25 hard negatives

Select high lexical matches that lack a positive label. DPR showed their value:
the model must distinguish same terms with wrong semantics. They are vulnerable
to incomplete qrels and answer-string false negatives.

### 4.5 ANN-mined negatives

[ANCE](https://openreview.net/forum?id=zeFrfgyZln) periodically embeds the
corpus with a current or recent encoder, builds ANN, and retrieves each query’s
nearest nonpositive documents. This targets the model’s active confusions.

Index refresh is asynchronous and expensive. If too stale, negatives no longer
match the model; if refreshed constantly, training stalls on re-encoding.
Record model/index generation and mining cadence.

### 4.6 Teacher-denoised negatives

[RocketQA](https://aclanthology.org/2021.naacl-main.466/) uses a cross-encoder
to filter noisy hard negatives and augment positives. A teacher can distinguish
same-answer or actually relevant candidates, but transfers its biases. Audit
teacher errors, especially on long, multilingual, temporal, and table evidence.

### 4.7 Adversarial and counterfactual negatives

Construct same-entity/wrong-relation, wrong-date/version, negated, unit-swapped,
source-spoofed, and nearly supporting passages. These train the distinctions RAG
needs. Generated counterfactuals must not accidentally remain true or become
unnatural shortcuts.

### 4.8 False-negative mitigation

- retrieve/judge alternative positives;
- mask duplicate or same-answer candidates;
- use soft teacher relevance instead of binary labels;
- use debiased contrastive losses;
- downweight ambiguous candidates;
- inspect hard-negative clusters manually;
- retain source/time/scope metadata in labels;
- evaluate against pooled, expanded qrels.

Harder is not always better: a false negative near the decision boundary creates
a large harmful gradient.

## 5. Dense retriever objectives

### 5.1 Multiple-negative softmax / InfoNCE

\[
\mathcal L_i=-\log
\frac{e^{s(q_i,d_i^+)/\tau}}
{e^{s(q_i,d_i^+)/\tau}+\sum_j e^{s(q_i,d_{ij}^-)/\tau}}.
\]

Temperature controls concentration. With dot product, model norms can change
effective temperature; L2 normalization removes that degree of freedom. Batch
composition and number of negatives materially change the loss.

### 5.2 Triplet and margin loss

\[
\mathcal L_i=\max(0,m-s(q_i,d_i^+)+s(q_i,d_i^-)).
\]

It focuses on violations but ignores already separated pairs and requires a
margin. Multiple hard negatives or smooth softplus variants improve training.

### 5.3 Pairwise logistic loss

\[
\mathcal L_i=\log(1+e^{-(s^+-s^-)}).
\]

This gives continuous gradients and is common in ranking/distillation.

### 5.4 Listwise likelihood

Normalize over a candidate list and match one-hot or graded relevance. Listwise
training better reflects ordering, but candidates and incomplete labels define
the target distribution.

### 5.5 Margin-MSE distillation

[GPL](https://aclanthology.org/2022.naacl-main.168/) matches teacher score
margins:

\[
\mathcal L=(s_\theta(q,d^+)-s_\theta(q,d^-)
-[s_T(q,d^+)-s_T(q,d^-)])^2.
\]

Margins carry graded preference without requiring comparable absolute teacher
and student scales. Teacher calibration and candidate sampling still matter.

### 5.6 KL/listwise distillation

For teacher distribution \(p_T(d\mid q)\) and student \(p_S\), minimize

\[
\operatorname{KL}(p_T\|p_S)=\sum_d p_T(d\mid q)
\log\frac{p_T(d\mid q)}{p_S(d\mid q)}.
\]

Temperature can expose dark knowledge among negatives. Candidate lists must
include meaningful alternatives; distillation cannot teach unseen distinctions.

## 6. Retrieval-oriented pretraining

### Inverse cloze task

Select a sentence as pseudo-query and its surrounding context as positive.
ORQA used ICT to overcome latent-retrieval cold start. It supplies massive weak
data but may teach document-local rather than user-query relevance.

### Condenser

[Condenser](https://aclanthology.org/2021.emnlp-main.75/) uses an architecture
that forces late MLM processing through a `[CLS]` representation, making the
bottleneck useful before retrieval fine-tuning.

### coCondenser

[coCondenser](https://aclanthology.org/2022.acl-long.203/) adds corpus-aware
contrastive learning between spans of the same document. It creates global
semantic structure without labeled queries.

### RetroMAE

[RetroMAE](https://aclanthology.org/2022.emnlp-main.35/) gives a full encoder a
lightly masked input and a shallow decoder a heavily masked view, forcing the
sentence embedding to preserve reconstructive information.

### SimLM

[SimLM](https://aclanthology.org/2023.acl-long.125/) uses a bottlenecked
replaced-language-modeling objective. Its goal is to make the single vector
carry token-level information relevant to retrieval.

### Weakly supervised pair pretraining

E5-style training collects many text-pair relations and uses large-batch
contrastive learning with explicit query/passage prefixes. INSTRUCTOR conditions
embeddings on task instructions. Generality depends on pair diversity,
instruction coverage, and contamination.

## 7. Learned sparse objectives

SPLADE-style systems optimize ranking loss plus query/document sparsity:

\[
\mathcal L=\mathcal L_{rank}
+\lambda_q\mathcal R(w(q))
+\lambda_d\mathcal R(w(d)).
\]

FLOPS regularization penalizes vocabulary dimensions frequently active over a
batch, approximating posting-list work. L1 penalizes each representation’s
mass. Query and document costs differ: query nonzeros affect lists opened;
document nonzeros affect index size/posting density.

Training choices include MLM initialization, hard negatives, cross-encoder
distillation, ensemble teachers, self-distillation, quantization, and curriculum
on regularization. Report effectiveness at matched index/latency budgets, not
only the best unconstrained model.

## 8. Multi-vector training

ColBERT optimizes passage ranking while retaining token vectors. MaxSim creates
hard discrete token alignments; in-batch negatives teach which local matches
matter. ColBERTv2 uses denoised supervision and residual compression. Later
systems train routing or token retrievability to reduce scoring cost.

Negative construction should include passages with overlapping keywords but
wrong relations, because MaxSim can overvalue isolated term matches. Training
must match the serving compression/index; quantization-aware or centroid-aware
distillation may reduce train/serve mismatch.

Visual late-interaction models train query-token/page-patch alignment from
document retrieval pairs, often with synthetic or weak labels. Evaluate domain
overlap, language, page-level ambiguity, layout degradation, and storage.

## 9. Reranker training

### Pointwise

Train binary/graded relevance per query-document pair. Easy to calibrate and
batch, but independent scores ignore list/set structure.

### Pairwise

Optimize positive over negative. Sample pairs across rank positions and
retriever sources. If training negatives come only from BM25, the model may not
learn dense-retriever errors.

### Listwise and Lambda losses

Optimize a list distribution or weight pairwise gradients by metric change,
such as \(|\Delta\mathrm{nDCG}|\). This aligns ranking metrics but qrel
incompleteness can heavily penalize unjudged valid evidence.

### Generative ranking

monoT5 generates a relevance token. RankT5 directly predicts scores. LLM
rerankers output permutations or pairwise preferences. Distill expensive
teachers into smaller cross-encoders, retaining hard candidate lists and teacher
uncertainty.

### Downstream-utility training

Relevance is not identical to usefulness. Train a selector using generator loss,
answer correctness, claim coverage, or marginal contribution. Control for
generator parametric knowledge: a passage may appear useless because the model
already knows the answer, or useful only to one generator.

## 10. Query rewriter and decomposer training

Supervision can be human standalone rewrites, teacher rewrites, clicked/relevant
documents, or downstream answer reward. Text-likelihood training copies
plausible rewrites but may not optimize retrieval. A retrieval-aware objective
maximizes relevant-document probability:

\[
\mathcal L_{rewrite}=-\log
\sum_{d\in D^+}p_\eta(d\mid \hat q_\phi(q,h)).
\]

Jointly training through a discrete search/query string is difficult; use policy
gradient, sequence-level distillation, or differentiable retriever proxies.

For decomposition, label subquestions, dependencies, required evidence, and
stop state. Penalize redundant/unanswerable subquestions and fan-out cost.
Always test intent preservation and direct-query fallback.

## 11. Generator supervised fine-tuning

Train on `(instruction, evidence, answer, citations)` examples with varied
evidence conditions:

- complete gold support;
- partial support;
- no support;
- irrelevant distractors;
- same-entity hard distractors;
- conflicts and stale/current versions;
- source-authority differences;
- malicious embedded instructions;
- long and reordered contexts.

If training always supplies perfect context, the model learns neither robust
selection nor abstention. If citations always appear at answer end, it will not
learn local attribution.

### Token likelihood

\[
\mathcal L_{gen}=-\sum_t\log p_\theta(y_t\mid y_{<t},q,Z).
\]

This rewards copying reference style and content. It does not separately reward
factuality, completeness, support, or citation validity.

### Evidence dropout and distractor curricula

Randomly remove supporting units, add negatives, change order, or vary evidence
budget. Teach the output state (`answer`, `partial`, `insufficient`, `conflict`)
and ensure removal does not create incorrectly labeled examples.

### Citation training

Use stable evidence IDs and attach them to atomic claims. Loss can include
citation tokens plus auxiliary support/alignment objectives. Negative examples
include wrong-but-retrieved IDs, non-supporting mentions, and fabricated IDs.
Validate citation existence deterministically at inference.

## 12. Latent-document joint training

RAG/REALM-style models optimize

\[
\mathcal L=-\log\sum_{z\in\operatorname{TopK}_\eta(q)}
p_\eta(z\mid q)p_\theta(y\mid q,z).
\]

The generator likelihood supplies a retriever signal: documents making the
answer likely gain probability. Problems:

- top-k truncation gives zero gradient to unretrieved evidence;
- generator may assign high likelihood to spurious answer-containing passages;
- parametric knowledge weakens evidence credit;
- document embeddings/index become stale;
- re-indexing is expensive and nondifferentiable;
- likelihood does not enforce citation or authority.

Warm-start with retrieval supervision/pretraining, refresh indexes, and combine
explicit evidence labels or distillation.

## 13. EM and multi-stage joint learning

Treat evidence as latent and alternate:

1. **E-like step:** estimate a posterior or reader utility over documents;
2. **M-like step:** train retriever and generator from that distribution.

EMDR² and related methods approximate end-to-end learning for multi-document
QA. Teacher posteriors can be sharp and unstable; evidence sets interact; the
candidate pool remains a ceiling. Log posterior entropy and whether mass falls
on genuinely supporting documents.

## 14. Reader-to-retriever distillation

FiD-KD transfers reader signals—attention or passage contribution—to a
retriever. Let teacher logits \(t_i\) and retriever logits \(s_i\):

\[
\mathcal L_{KD}=\operatorname{KL}
(\operatorname{softmax}(t/T)\|\operatorname{softmax}(s/T)).
\]

Attention is not guaranteed causal evidence use. Alternative leave-one-out
utility measures the answer-likelihood change after removing a passage, at
greater cost. Distillation is tied to reader, prompt, and candidate set.

## 15. Preference optimization for grounded generation

Construct preferred/rejected answers or trajectories differing in correctness,
support, completeness, citation, abstention, and cost. Direct Preference
Optimization uses

\[
\mathcal L_{DPO}=-\log\sigma\left(
\beta\log\frac{\pi_\theta(y^+\mid x)}{\pi_{ref}(y^+\mid x)}
-\beta\log\frac{\pi_\theta(y^-\mid x)}{\pi_{ref}(y^-\mid x)}
\right).
\]

Pairs should isolate the desired property. If preferred answers are also longer
or more stylistically polished, the model learns that shortcut. Include
faithful concise and unfaithful fluent contrasts, and audit by claim.

## 16. Reinforcement learning for search

Model retrieval control as an MDP:

- state \(s_t\): request, reasoning state, selected evidence, remaining budget;
- action \(a_t\): no-search, query, retriever/source, select, read, answer, stop;
- transition: search/tool result and model state update;
- reward: correctness, support, citation, process quality, latency/cost/risk;
- terminal condition: answer, abstain, or hard budget.

### Policy gradient

\[
\nabla J(\theta)=
\mathbb E_\tau\left[
\sum_t\nabla\log\pi_\theta(a_t\mid s_t)(R(\tau)-b_t)
\right].
\]

PPO clips policy-ratio changes; GRPO-style methods use group-relative
advantages without a learned critic; DAPO and variants modify sampling and
optimization. The algorithm name is less important than action validity,
reward definition, exploration, and corpus/retriever setup.

### Outcome rewards

Exact match/F1 or an answer judge is easy but cannot prove good search. The model
may answer from memory, fabricate retrieval tags, exploit formatting, or use
irrelevant passages. ReSearch and Search-R1 demonstrate strong outcome-trained
search while retaining these attribution caveats.

### Process rewards

Reward useful search timing, information gain, nonredundancy, evidence support,
and correct stop. StepSearch uses stepwise process signals; HiPRAG defines
hierarchical rewards based on optimal search decisions. Process labels can be
synthetic or task-specific and may not transfer.

### Value-based evidence selection

Q-RAG freezes the LLM and learns an embedding-based Q-function over candidate
selection/STOP actions. It separates retriever-policy learning from generator
weights and can scale selection, but relies on support-fact supervision and its
defined terminal reward.

### Cost and risk rewards

Use explicit constraints or Lagrangian penalties:

\[
R=R_{quality}+\alpha R_{support}
-\lambda_1N_{search}-\lambda_2\text{tokens}
-\lambda_3\text{latency}-\lambda_4\text{risk}.
\]

Average penalties do not guarantee per-request limits. Enforce maximum calls,
source permissions, and timeouts outside the policy.

## 17. Reward design and hacking tests

Separate reward components and log each:

- answer correctness;
- completeness;
- citation entailment and completeness;
- evidence-set recall/authority/freshness;
- calibrated abstention;
- valid query/action syntax;
- unique useful evidence and information gain;
- step/call/token/latency/cost;
- safety and policy compliance.

Red-team the reward:

- fabricated `<search>`/document tags;
- answering without searching while claiming retrieval;
- copying reference phrases;
- over-search to accumulate process points;
- trivial stop to avoid cost;
- citation to any retrieved document regardless of support;
- judge-prompt injection from evidence;
- length/style shortcuts;
- exploiting known benchmark answer formats.

Use held-out judges, human audits, counterfactual evidence swaps, and tests where
parametric knowledge conflicts with documents.

## 18. Curriculum and staged training

A practical sequence is:

1. retrieval-oriented encoder pretraining or strong pretrained embeddings;
2. supervised/weak retriever training with in-batch and lexical negatives;
3. ANN mining and teacher denoising;
4. reranker training on the deployed hybrid candidate distribution;
5. generator SFT with complete/partial/noisy/conflicting evidence and citations;
6. reader-to-retriever or utility distillation;
7. supervised search/control trajectories;
8. preference/RL optimization under hard budgets;
9. domain calibration and human evaluation;
10. continual mining/retraining with drift and rollback controls.

Jointly optimizing everything from random initialization is rarely stable or
necessary. Keep strong modular baselines to locate gains.

## 19. Domain adaptation

### Retriever

- continue MLM/retrieval pretraining on unlabeled domain text;
- generate domain queries and teacher margins (GPL-style);
- collect human queries/qrels;
- mine hard negatives from the target corpus;
- preserve general-domain mixtures to reduce catastrophic forgetting;
- tune analyzers and retain BM25/hybrid baselines.

### Generator

- teach domain terminology and output schemas;
- include evidence authority/time rules and abstention;
- use retrieved context conditions matching deployment;
- keep evaluation separate from synthetic teacher generation.

### Specialized domains

Medical/legal/financial models require authoritative-source and temporal labels,
not merely domain vocabulary. Code retrievers need repository structure and
revision-aware negatives. Multilingual adaptation needs language-balanced
batches and cross-lingual positives; English teacher quality can create unequal
performance.

## 20. Continual learning and index consistency

Corpus change creates new positives, negatives, terms, entities, and embedding
distribution. Monitor:

- retrieval score/rank distributions;
- new-query and no-result rate;
- language/domain/tenant mix;
- qrel and answer quality by time;
- hard-negative composition;
- embedding norms and ANN recall;
- stale-answer and superseded-source rate.

Retraining changes the vector space. Build a new index generation, evaluate
paired traffic, and cut over atomically. Online query-encoder updates against
old document embeddings are unsafe unless compatibility is explicitly trained
and verified.

Avoid feedback loops where only retrieved/shown/clicked documents become future
positives. Maintain exploration, diverse judging pools, and counterfactual
evaluation.

## 21. Training security and privacy

- remove secrets/PII or use approved local processing;
- preserve consent/license/training-eligibility metadata;
- defend against poisoned documents and synthetic labels;
- isolate untrusted text from teacher/system prompts;
- track source lineage into training examples;
- support deletion/unlearning policy across datasets/checkpoints where promised;
- audit memorization and corpus extraction;
- prevent cross-tenant examples and batch negatives;
- secure model, index, and trajectory artifacts.

RAG’s external datastore can reduce the need to bake private facts into weights,
but training on retrieved or logged private content recreates the problem.

## 22. Evaluation protocol for training claims

### Frozen component evaluations

- retriever with fixed corpus/chunker/index;
- reranker on a frozen candidate pool;
- generator with gold and controlled contexts;
- policy with a frozen retriever/generator/tool environment.

### End-to-end evaluation

After component gains, rebuild the complete pipeline and measure answer,
citations, abstention, latency, cost, and risk. A retriever nDCG gain can add
longer/noisier contexts and reduce generation quality.

### Statistical design

- paired per-query comparisons;
- confidence intervals/bootstrap or randomization tests;
- multiple seeds for stochastic training and RL;
- versioned model/index/corpus/prompts;
- contamination checks;
- slice metrics and worst-group behavior;
- human audit of high-impact disagreements;
- compute/energy and training/inference resource report.

### Ablations

Remove pretraining, negative types, teacher denoising, distillation, process
rewards, cost penalty, citations, and curriculum stages individually. Report the
candidate/index setup for each. If several changes enter together, the source of
improvement is unknown.

## 23. Choosing what to train

| Constraint/problem | First training target | Why |
|---|---|---|
| No labels, broad domain | strong general embedding + BM25; synthetic/GPL adaptation | avoids premature custom model |
| Vocabulary/domain mismatch | retriever/analyzer with target qrels | directly improves recall ceiling |
| High first-stage recall, poor precision | reranker/selector on deployed candidates | cheaper than retraining corpus embeddings |
| Evidence present but ignored | generator SFT with noisy/partial contexts | retrieval is not the bottleneck |
| Missing multi-hop evidence | decomposer/search policy + set supervision | one-shot ranker objective is wrong |
| Over-retrieval/cost | retrieve/stop policy with quality-cost objective | adapt budget by query |
| Citation errors | claim/evidence SFT and verifier | answer correctness alone insufficient |
| Domain shift | domain queries, hard negatives, calibration | benchmark-general model may not transfer |
| Frequent corpus change | modular retriever/index + continual evaluation | joint retraining may be too slow |
| High privacy/regulation | external governed memory; minimal private fine-tuning | improves deletion/audit boundary |

## 24. What the executable notebooks model

The training notebook computes multiple-negative, margin, pairwise, listwise,
distillation, DPO, and policy-gradient objectives on inspectable examples. It
demonstrates random versus in-batch versus lexical/ANN hard negatives, false
negative masking, teacher soft labels, reward decomposition, and cost-constrained
search trajectories. It also shows why a correct answer can assign the wrong
retriever credit.

No notebook trains a billion-parameter model. The objective is to make the
mathematics, label assumptions, and failure modes executable before substituting
real encoders, LLMs, distributed indexes, or RL infrastructure.