RAG technique decision guide

This guide converts the literature into an experiment plan. It assumes the evaluation protocol in evaluation_and_risks.md and does not treat a framework or vector database as an architecture.

1. Start from the information task

2. A production reference architecture

                           ┌──────────────────────────────────────┐
sources → parse/version →  │ sparse │ dense │ visual │ graph/API │
          ACL/time/hash    └─────── candidate retrieval ──────────┘
                                           │
query → intent/permission/time → rewrite/decompose/route
                                           │
                              union + RRF/calibration
                                           │
                        high-precision rerank / utility score
                                           │
                   dedupe + conflict detection + context budget
                                           │
                 evidence envelope (untrusted data, immutable IDs)
                                           │
                     generate / iterate / abstain / escalate
                                           │
                  atomic claims ↔ exact source spans + versions
                                           │
          trace, evaluate, monitor, feedback, correction, deletion

Ingestion plane

For each source object, retain:

• canonical document and version ID;
• content hash, owner, tenant, ACL, license, retention/deletion status;
• event time, valid time, ingestion time, and source trust tier;
• parser version, page/section/character offsets, and modality;
• lineage from derived chunk, proposition, summary, graph node, or image patch to the original bytes.

Do not create only embeddings. A vector without source/version/permission metadata cannot support citations, corrections, access control, or deletion.

Candidate retrieval plane

Use heterogeneous retrieval for different error surfaces:

• BM25 or learned sparse for rare and exact lexical evidence;
• a domain-tested dense encoder for paraphrase/semantic recall;
• late interaction when token-level matching justifies memory/latency;
• visual page retrieval for layout/figure/table evidence;
• structured SQL/API filters for hard constraints and current records;
• graph/proposition traversal only for validated relationship tasks.

Fuse rankings with reciprocal-rank fusion when score distributions are not calibrated:

Score normalization can outperform RRF when learned on representative qrels, but it is more vulnerable to drift. Preserve each component rank and raw score for diagnosis.

Reranking and evidence utility

A cross-encoder or late-interaction reranker should see the exact query and candidate. Train/evaluate with realistic hard negatives: topically similar but logically useless, stale, conflicting, wrong-tenant, duplicate, and adversarial documents. Optimize downstream evidence utility under a budget, not only relevance.

Reranker evaluation must include first-stage oracle recall. If gold evidence is absent from candidates, the reranker cannot recover it.

Context construction

Before generation:

1. remove exact and near duplicates;
2. enforce permissions again;
3. group versions and expose conflicts;
4. choose diverse evidence with MMR or a learned utility model;
5. preserve local surrounding context and document structure;
6. place strongest evidence where the reader uses it reliably;
7. stay below the empirically optimal token budget, not the maximum window;
8. wrap evidence as untrusted data with immutable IDs.

A generic MMR selection step is

Generation and control policy

Begin with one-shot generation and explicit abstention. Add policy complexity only for a measured failure:

• query rewriting for conversation/contextualization;
• multi-query or HyDE for vocabulary mismatch;
• correction/web fallback for low-confidence retrieval;
• decomposition/iterative retrieval for multi-hop recall;
• a graph/hierarchy for relational or global abstraction;
• long-context fallback when retrieval says evidence is insufficient;
• learned retrieve/stop actions when fixed policies waste budget or miss steps.

Every loop needs hard maximum tool calls, wall time, tokens, cost, source scope, and a termination/abstention condition. Outcome RL without process and evidence checks can learn reward-hacking trajectories.

Attribution

Generate or post-process atomic claims. Each claim maps to source ID + exact span/page/region + version. Reject invented IDs. Run completeness and entailment checks separately; treat model/NLI scores as fallible. Show conflicting or stale sources. High-stakes claims require human review.

3. The experiment ladder

Each rung must beat the prior one on a paired held-out set under hard safety and permission gates.

1. Closed-book generator — establishes parametric knowledge and leakage.
2. BM25 top-\(k\) — strong, cheap lexical baseline.
3. Domain dense top-\(k\) — isolates semantic retrieval.
4. Sparse+dense RRF — tests complementary recall.
5. Reranking — tests precision and evidence utility.
6. Deduplication/context budgeting/order — tests reader distraction.
7. Claim citations + abstention — establishes trust behavior.
8. Query rewrite/multi-query — only for observed query mismatch.
9. Iterative/graph/hierarchical/visual/long-context route — select the one matching a tagged error slice.
10. Learned policy or agent — only when the fixed strategy's quality/cost frontier is inadequate.

For each rung save per-query candidates and deltas. An average gain that comes only from one dataset slice may justify routing rather than replacing the whole pipeline.

4. Routing policy examples

These are initial hypotheses to validate, not universal rules.

Train a router only after the action set is proven and labeled. Measure calibration, confusion costs, policy drift, and oracle action gaps.

5. Deployment gates

Correctness and grounding

• product-gold retrieval claim recall and context precision meet thresholds;
• oracle-context generator proves it can use correct evidence;
• end-to-end claim correctness, completeness, and citation entailment pass by slice;
• answerable/unanswerable false-answer and false-abstention rates pass;
• counterfactual, conflict, noise, and evidence-order tests pass.

Security and permissions

• pre- and post-retrieval ACL enforcement and tenant-isolation tests pass;
• poison, indirect instruction, ad, DoS, and exfiltration red-team cases pass;
• retrieved evidence cannot invoke tools or change system instructions;
• sources, hashes, actions, and model decisions are auditable;
• deletion removes content from every derived store and cache.

Freshness and operations

• ingestion/update/deletion SLAs are measured and monitored;
• current queries use temporal constraints and authoritative sources;
• a frozen corpus/query snapshot can replay evaluations;
• p95 latency, peak memory, throughput, and cost/supported-answer meet budgets;
• fallback behavior survives retriever, index, model, and external-search outage.

6. Common anti-patterns

• Selecting an embedding from a global MTEB mean without domain qrels.
• Calling a vector-store demo “RAG” while omitting reranking, provenance, evaluation, permission, and update design.
• Using only vector recall or only answer accuracy.
• Increasing \(k\) because recall rises, without measuring reader distraction.
• Reporting LLM-as-judge scores without judge/version/prompt and human audit.
• Treating a cited URL as proof of entailment or authority.
• Building a global graph before showing a relational/global task gap.
• Replacing retrieval with long context based on nominal window size.
• Claiming freshness while serving an unmonitored batch index or stale cache.
• Letting retrieved text share an instruction channel or tool permissions.
• Training search agents on answer reward without evidence/process/cost checks.
• Comparing paper headline numbers across different corpora, readers, and context budgets.

7. A practical 2026 default

For a typical text knowledge base:

1. versioned structure-aware chunks with ACL/time/source metadata;
2. BM25 + a validated dense encoder;
3. RRF candidate union;
4. a domain-tested cross-encoder reranker;
5. deduplication, conflict detection, and a tuned evidence-token budget;
6. a conservative prompt treating evidence as untrusted data;
7. atomic claim citations and evidence-insufficient abstention;
8. a trace storing candidates, scores, versions, spans, latency, and cost;
9. a layered gold evaluation with paired uncertainty and red-team gates;
10. routing to specialized graph, visual, iterative, live, or long-context methods only for the slices where they win.

That architecture is modular enough to incorporate the frontier while keeping each new technique accountable to a specific measured failure.