# RAG technique decision guide This guide converts the literature into an experiment plan. It assumes the evaluation protocol in [evaluation_and_risks.md](evaluation_and_risks.md) and does not treat a framework or vector database as an architecture. ## 1. Start from the information task | Task shape | Start here | Add only if evidence shows a gap | Do not assume | |---|---|---|---| | Rare identifiers, error codes, named clauses | BM25/learned sparse; metadata filters | dense fusion; cross-encoder | dense embeddings preserve exact strings | | Semantic FAQ/fact lookup | sparse+dense union, RRF, reranker | query rewriting/HyDE on missed paraphrases | a larger \(k\) improves answers | | Multi-hop question | hybrid candidates; query decomposition; iterative retrieval | proposition/local graph paths; learned search policy | a static top-\(k\) contains a complete reasoning chain | | Global corpus synthesis | sampled/map-reduce baseline | community summaries or hierarchical index | GraphRAG improves local fact lookup | | Long-document QA | structure-aware chunks; rerank; evidence ordering | RAPTOR-style hierarchy; route to full context | advertised context length equals effective use | | Tables, figures, scanned PDFs | OCR/layout baseline plus page-image retrieval | ColPali/VisRAG/VDocRAG; visual reranker | visual-only or OCR-only wins every document type | | Dynamic web/current facts | source-specific APIs/live search, temporal filters, snapshots | corrective fallback and contradiction handling | retrieval makes stale caches fresh | | Multi-turn support | turn-aware rewrite and flat history baseline | multi-granular memory and consolidation | nearest old turn captures user intent or updates | | High-stakes cited answer | trusted corpus, claim/span IDs, abstention, human review | claim-guided search, NLI/judge as secondary check | citations prove correctness or authority | | Tool-using agent | read-only retrieval in a sandbox; explicit action approval | learned search/stop policy with hard budgets | retrieved text is safe to execute | ## 2. A production reference architecture ```text ┌──────────────────────────────────────┐ sources → parse/version → │ sparse │ dense │ visual │ graph/API │ ACL/time/hash └─────── candidate retrieval ──────────┘ │ query → intent/permission/time → rewrite/decompose/route │ union + RRF/calibration │ high-precision rerank / utility score │ dedupe + conflict detection + context budget │ evidence envelope (untrusted data, immutable IDs) │ generate / iterate / abstain / escalate │ atomic claims ↔ exact source spans + versions │ trace, evaluate, monitor, feedback, correction, deletion ``` ### Ingestion plane For each source object, retain: - canonical document and version ID; - content hash, owner, tenant, ACL, license, retention/deletion status; - event time, valid time, ingestion time, and source trust tier; - parser version, page/section/character offsets, and modality; - lineage from derived chunk, proposition, summary, graph node, or image patch to the original bytes. Do not create only embeddings. A vector without source/version/permission metadata cannot support citations, corrections, access control, or deletion. ### Candidate retrieval plane Use heterogeneous retrieval for different error surfaces: - BM25 or learned sparse for rare and exact lexical evidence; - a domain-tested dense encoder for paraphrase/semantic recall; - late interaction when token-level matching justifies memory/latency; - visual page retrieval for layout/figure/table evidence; - structured SQL/API filters for hard constraints and current records; - graph/proposition traversal only for validated relationship tasks. Fuse rankings with reciprocal-rank fusion when score distributions are not calibrated: \[ \operatorname{RRF}(d)=\sum_r \frac{w_r}{K+\operatorname{rank}_r(d)}. \] Score normalization can outperform RRF when learned on representative qrels, but it is more vulnerable to drift. Preserve each component rank and raw score for diagnosis. ### Reranking and evidence utility A cross-encoder or late-interaction reranker should see the exact query and candidate. Train/evaluate with realistic hard negatives: topically similar but logically useless, stale, conflicting, wrong-tenant, duplicate, and adversarial documents. Optimize downstream evidence utility under a budget, not only relevance. Reranker evaluation must include first-stage oracle recall. If gold evidence is absent from candidates, the reranker cannot recover it. ### Context construction Before generation: 1. remove exact and near duplicates; 2. enforce permissions again; 3. group versions and expose conflicts; 4. choose diverse evidence with MMR or a learned utility model; 5. preserve local surrounding context and document structure; 6. place strongest evidence where the reader uses it reliably; 7. stay below the empirically optimal token budget, not the maximum window; 8. wrap evidence as untrusted data with immutable IDs. A generic MMR selection step is \[ d^*=\arg\max_{d\notin S} \lambda\operatorname{rel}(q,d) -(1-\lambda)\max_{s\in S}\operatorname{sim}(d,s). \] ### Generation and control policy Begin with one-shot generation and explicit abstention. Add policy complexity only for a measured failure: - query rewriting for conversation/contextualization; - multi-query or HyDE for vocabulary mismatch; - correction/web fallback for low-confidence retrieval; - decomposition/iterative retrieval for multi-hop recall; - a graph/hierarchy for relational or global abstraction; - long-context fallback when retrieval says evidence is insufficient; - learned retrieve/stop actions when fixed policies waste budget or miss steps. Every loop needs hard maximum tool calls, wall time, tokens, cost, source scope, and a termination/abstention condition. Outcome RL without process and evidence checks can learn reward-hacking trajectories. ### Attribution Generate or post-process atomic claims. Each claim maps to source ID + exact span/page/region + version. Reject invented IDs. Run completeness and entailment checks separately; treat model/NLI scores as fallible. Show conflicting or stale sources. High-stakes claims require human review. ## 3. The experiment ladder Each rung must beat the prior one on a paired held-out set under hard safety and permission gates. 1. **Closed-book generator** — establishes parametric knowledge and leakage. 2. **BM25 top-\(k\)** — strong, cheap lexical baseline. 3. **Domain dense top-\(k\)** — isolates semantic retrieval. 4. **Sparse+dense RRF** — tests complementary recall. 5. **Reranking** — tests precision and evidence utility. 6. **Deduplication/context budgeting/order** — tests reader distraction. 7. **Claim citations + abstention** — establishes trust behavior. 8. **Query rewrite/multi-query** — only for observed query mismatch. 9. **Iterative/graph/hierarchical/visual/long-context route** — select the one matching a tagged error slice. 10. **Learned policy or agent** — only when the fixed strategy's quality/cost frontier is inadequate. For each rung save per-query candidates and deltas. An average gain that comes only from one dataset slice may justify routing rather than replacing the whole pipeline. ## 4. Routing policy examples These are initial hypotheses to validate, not universal rules. | Observable query/corpus signal | Candidate action | |---|---| | quoted phrase, SKU, code, statute, proper noun | boost sparse/exact and metadata fields | | paraphrase or conceptual description | increase dense weight; consider query expansion | | comparison, temporal chain, “why/how connected” | decompose; retrieve each subquestion; local graph expansion | | “overall themes/trends across corpus” | global summaries/community reports or map-reduce | | table/chart/page reference | visual+OCR/layout hybrid retrieval | | current/latest/live | authoritative API/live index with temporal validity; never static memory alone | | answer missing after strong retrieval | abstain, broaden source, or full-context fallback—not hallucinate | | strong conflicting sources | preserve both, rank authority/time, state conflict, escalate if high stakes | | low-risk easy question with confident parametric answer | possibly no retrieval, but audit freshness and citation requirement | Train a router only after the action set is proven and labeled. Measure calibration, confusion costs, policy drift, and oracle action gaps. ## 5. Deployment gates ### Correctness and grounding - product-gold retrieval claim recall and context precision meet thresholds; - oracle-context generator proves it can use correct evidence; - end-to-end claim correctness, completeness, and citation entailment pass by slice; - answerable/unanswerable false-answer and false-abstention rates pass; - counterfactual, conflict, noise, and evidence-order tests pass. ### Security and permissions - pre- and post-retrieval ACL enforcement and tenant-isolation tests pass; - poison, indirect instruction, ad, DoS, and exfiltration red-team cases pass; - retrieved evidence cannot invoke tools or change system instructions; - sources, hashes, actions, and model decisions are auditable; - deletion removes content from every derived store and cache. ### Freshness and operations - ingestion/update/deletion SLAs are measured and monitored; - current queries use temporal constraints and authoritative sources; - a frozen corpus/query snapshot can replay evaluations; - p95 latency, peak memory, throughput, and cost/supported-answer meet budgets; - fallback behavior survives retriever, index, model, and external-search outage. ## 6. Common anti-patterns - Selecting an embedding from a global MTEB mean without domain qrels. - Calling a vector-store demo “RAG” while omitting reranking, provenance, evaluation, permission, and update design. - Using only vector recall or only answer accuracy. - Increasing \(k\) because recall rises, without measuring reader distraction. - Reporting LLM-as-judge scores without judge/version/prompt and human audit. - Treating a cited URL as proof of entailment or authority. - Building a global graph before showing a relational/global task gap. - Replacing retrieval with long context based on nominal window size. - Claiming freshness while serving an unmonitored batch index or stale cache. - Letting retrieved text share an instruction channel or tool permissions. - Training search agents on answer reward without evidence/process/cost checks. - Comparing paper headline numbers across different corpora, readers, and context budgets. ## 7. A practical 2026 default For a typical text knowledge base: 1. versioned structure-aware chunks with ACL/time/source metadata; 2. BM25 + a validated dense encoder; 3. RRF candidate union; 4. a domain-tested cross-encoder reranker; 5. deduplication, conflict detection, and a tuned evidence-token budget; 6. a conservative prompt treating evidence as untrusted data; 7. atomic claim citations and evidence-insufficient abstention; 8. a trace storing candidates, scores, versions, spans, latency, and cost; 9. a layered gold evaluation with paired uncertainty and red-team gates; 10. routing to specialized graph, visual, iterative, live, or long-context methods only for the slices where they win. That architecture is modular enough to incorporate the frontier while keeping each new technique accountable to a specific measured failure.