Production RAG systems: architecture, serving, observability, cost, and operations

A research pipeline that answers a benchmark question is not yet a production RAG system. Production adds continuous ingestion, versioned indexes, identity, tail latency, concurrency, cache invalidation, partial failure, rollout, monitoring, cost control, privacy, and incident response. This chapter turns the method taxonomy into an operating model.

1. Separate the control, data, and evaluation planes

Data plane

Processes requests:

gateway/auth -> policy/query analysis -> retrieval fan-out -> fusion/rerank
 -> evidence selection/context -> generator/tools -> verification/policy
 -> response/citations

Ingestion/index plane

Builds knowledge:

connectors/CDC -> raw objects -> parse/OCR -> normalize/dedup -> enrich/chunk
 -> sparse/vector/graph/table indexes -> validation -> immutable release

Control plane

Stores versioned configuration, feature flags, routing policy, source registry, index aliases, credentials/scopes, budgets, and rollout/rollback state.

Evaluation plane

Replays frozen datasets and production traces against candidate versions, computes component/end-to-end metrics, human audits, adversarial tests, and release reports. It must not mutate production or leak evaluation labels into online decisions.

Keeping these planes distinct prevents a user query from triggering an unsafe index mutation or a live config change from making experiments irreproducible.

2. Request lifecycle

authenticate principal and tenant;
apply rate limit and request policy;
assign request/trace ID and deadline;
classify language, task, risk, and freshness;
resolve ACL/policy snapshot;
construct original and transformed queries;
route/fan out to authorized indexes/tools;
fuse/deduplicate and rerank;
select/expand/compress evidence under budget;
generate or execute structured tools;
validate schema, source IDs, citations, safety, and DLP;
answer/partial/abstain/fallback;
record privacy-minimized trace and metrics;
collect feedback under explicit policy.

Propagate one absolute deadline. Each component receives the remaining budget and must cancel work after timeout. Independent retrievals can run concurrently; iterative search is sequential and therefore expensive at the tail.

3. Service-level indicators and objectives

Quality

answer correctness/completeness;
claim faithfulness and citation precision/recall;
abstention risk/coverage;
retrieval claim recall and context precision;
source authority/freshness;
task success/human resolution;
high-risk failure and policy violation.

Latency

Measure p50/p95/p99 separately:

gateway/auth/policy;
query rewrite/router;
each retriever and fan-out join;
fusion/rerank;
context/compression;
time to first token and generation;
tool/verification;
end-to-end.

Averages hide tail behavior. Report cold/warm cache, concurrency, payload size, and timeout/fallback rate.

Availability

Successful response should distinguish full answer, safe partial answer, abstention, and technical failure. A fluent hallucination is not availability. Track dependency availability and degraded-mode quality.

Freshness

Source-to-index lag, replica convergence, cache age, stale-answer rate, and current-version selection.

Security/privacy

ACL denials/violations, cross-tenant canaries, injection/poison anomalies, sensitive-output blocks, deletion SLA, and audit-log health.

Cost

Cost per request and per successful correctly cited answer, split by embedding, search, rerank, prompt/completion tokens, tools, verification, storage, and ingestion amortization.

4. Capacity model

Let arrival rate be $\lambda$ , service rate per worker $\mu$ , and workers $c$ . Keep utilization below saturation because queueing tail latency rises nonlinearly. Model each bottleneck separately: retriever CPU/SSD, reranker GPU, LLM prefill/decode, external APIs, and ingestion.

For a dense index:

S_{vec}=Nvdbr,

where $N$ units, $v$ vectors/unit, $d$ dimensions, $b$ bytes/value, and $r$ replicas. Add graph edges, ANN overhead, metadata, and source text.

For generation, prefill work grows with input tokens and model architecture; decode cost grows with output length and active sequences. RAG often increases prefill and reduces factual retries; model both.

Capacity tests use realistic query/corpus/filters, concurrency, batch shape, context/output lengths, cache state, and dependency latency. A single-query microbenchmark is not a capacity plan.

5. Retrieval serving

Sparse service

Shard by document/term/tenant depending workload. Use postings compression, WAND/block-max pruning, segment merges, filter indexes, and result caches. Monitor postings decoded, heap threshold, shard skew, hot terms, and merge pressure.

Dense ANN service

Choose exact/IVF/PQ/HNSW/DiskANN/ScaNN based on corpus, dimension, filters, memory, update rate, and SLO. Tune against exact neighbors and qrels. Monitor index generation, vector count/norms, ANN recall samples, graph/list health, filter selectivity, tombstones, and cache residency.

Multi-vector service

Track vectors/page or passage, centroid/posting skew, residual storage, candidate-stage recall, decompression, MaxSim compute, and GPU/CPU transfer. Route or rerank-only modes can control cost.

Federated fan-out

Search sources concurrently under per-source deadlines. Each response includes source/index generation, scores/ranks, partial/error state, and latency. Fusion must handle missing retrievers deterministically. Do not silently expand to an unauthorized fallback source.

6. Sharding and replication

Document sharding

Each shard contains a corpus partition; query all/selected shards and merge top-k. Uniform document count does not guarantee uniform posting/vector/search load. Tenant/domain/time partitions can improve filters but produce skew.

Term/posting sharding

Can distribute large inverted indexes but requires coordination for scoring. Most systems use document/segment partitions for simpler top-k merging.

Routing

Use tenant, language, domain, time, source type, or learned centroid routing. Router misses are a new recall ceiling; retain multi-shard fallback for uncertain queries.

Replicas

Replicas support availability/read throughput but create convergence and cache consistency issues. Attach generation ID to every result and avoid mixing incompatible generations. Test node loss and stale replicas.

7. Batching and scheduling

Embeddings

Batch by token count and model/input type. Separate query and document priorities; ingestion should not starve online queries. Cache repeated public queries only under safe policy.

Reranking

Batch query-document pairs with length bucketing. Candidate depth and truncation drive latency. Use dynamic batching with maximum wait and deadline awareness.

Generation

Continuous batching improves throughput; prompt length variation and long outputs create head-of-line effects. Prefix/prompt caching may save repeated system/evidence prefixes but has privacy and invalidation implications.

Priority and backpressure

Protect interactive/high-risk verification from bulk ingestion/evaluation. Use bounded queues, admission control, cancellation, and graceful degradation. Do not let retry storms multiply load.

8. Context and token budgets

Define budgets by request class:

query rewrite tokens/calls
retrieval calls and candidates per source
rerank pairs/tokens
selected evidence tokens/images/pages
generation input/output tokens
verification calls/tokens
total cost and wall-clock deadline

Routers can allocate budgets, but hard caps are enforced. Log planned and actual usage. Optimize cost per correct/cited answer; reducing retrieval tokens can increase hallucination or expensive retries.

Context packing should be deterministic given candidates/config, exposing items dropped for tokens, duplicates, source caps, trust, or coverage. Unexpected tokenizer/model changes can overflow limits; record exact tokenizer.

9. Caching layers

Source and parse cache

Key by content hash and parser version. Safe for immutable bytes; invalidate derived outputs when parser/enrichment changes.

Embedding cache

Key by normalized exact input hash, model/tokenizer/instruction/pooling/version. Sensitive text may require tenant-local encrypted storage or no cache.

Retrieval cache

Key by query representation, filters/ACL scope, index generation, k, and retriever config. Short TTL for volatile facts; invalidate on generation switch.

Semantic/answer cache

High risk: near queries may differ in tenant, date, intent, or constraints. Require strong isolation and verification. Store source/index/model versions and revalidate citations/freshness. Do not share private answer caches.

Prompt/KV cache

Can reduce LLM prefill for repeated prefixes/evidence. It consumes memory and can leak timing or cross-user content. Scope and audit carefully.

Cache evaluation

Track hit rate, correct-hit rate, age, stale-hit rate, saved latency/cost, memory, and privacy incidents. A high hit rate with wrong/stale responses is harmful.

10. Generation serving

Model selection and routing

Route by task/risk/context/modality/budget. Smaller models may handle extraction or classification; larger/VLM models handle synthesis or visual evidence. Keep fallback behavior and quality gates explicit.

Prefill/decode

RAG adds long evidence prefill. Context compression, evidence selection, prefix caching, quantization, speculative decoding, and efficient attention can help. Separate model kernel speed from end-to-end retrieval/tool latency.

Speculative retrieval

RaLMSpec speculates future retrievals and verifies them in batches for iterative retrieval-augmented LMs, reporting workload-dependent speedups without changing semantic outputs. It is most relevant when retrieval timing/query can be predicted and verification is cheaper than sequential waiting.

Structured output

Use schema-constrained responses for claims/evidence IDs/tool results, then validate deterministically. Invalid outputs trigger a bounded repair/fallback, not unlimited retries.

11. External dependencies and resilience

For each model/search/tool API define:

timeout and cancellation;
retry conditions with exponential backoff/jitter;
idempotency for writes;
circuit breaker;
rate/quota handling;
fallback and quality impact;
data transfer/privacy policy;
version-change detection;
cost ceiling.

Do not retry deterministic invalid requests or policy denials. Retries consume deadline and can duplicate tool actions. Capture partial retrieval results and answer only if policy/sufficiency permits.

12. Degraded modes

Examples:

dense index unavailable -> BM25 only with lower-confidence/coverage policy;
reranker unavailable -> simple fusion and smaller answer scope;
LLM unavailable -> search results/extractive answer;
web/API timeout -> static evidence with explicit as-of limitation;
graph unavailable -> hybrid text retrieval;
verification unavailable -> abstain/high-risk human review;
stale index -> block volatile queries or disclose snapshot time.

Test degraded modes regularly. A fallback that was never evaluated is not resilience.

13. Tracing and event schema

An end-to-end trace needs stable stages:

request_received, policy_resolved, query_classified, query_transformed,
retrieval_started/completed, fusion_completed, rerank_completed,
evidence_selected/compressed, generation_started/first_token/completed,
verification_completed, response_emitted, feedback_recorded

Each event includes version IDs, counts, latency, cost, result/evidence opaque IDs, decision reason, and error state. Content fields are minimized/redacted and access-controlled. Sampling should retain rare failures/high-risk slices without over-logging private data.

Distributed trace context must survive fan-out, tools, and retries. Otherwise tail latency and answer provenance cannot be reconstructed.

14. Online monitoring

Data/index health

connector lag/failures and source reconciliation;
parser/OCR confidence and document-type shifts;
chunk/vector/posting counts and distribution;
embedding norms/NaNs and duplicate rate;
index replica generation and ANN recall sample;
tombstone/delete backlog;
authority/language/domain/time composition.

Retrieval behavior

no-result and low-score rate;
score/rank entropy and gaps;
sparse/dense overlap and unique contribution;
candidate/selected/cited survival;
filter selectivity and result count;
duplicate/source concentration;
route/call/loop/fallback rates.

Generation behavior

answer/partial/abstain/error;
claim/citation count and invalid IDs;
sampled faithfulness/correctness;
output length, refusal, schema failure;
conflict/freshness disclosure;
human feedback/resolution.

Systems/economics

latency/throughput/saturation by stage;
token/tool calls and cost;
cache correct-hit/stale-hit;
retries/timeouts/circuit breakers;
cost per successful cited answer;
error budget burn.

15. Quality monitoring without immediate labels

Production correctness labels arrive slowly. Use leading indicators cautiously:

retrieval score/overlap/drift;
citation-ID validity/accessibility;
deterministic number/date/entity consistency;
support judge calibrated on human labels;
answer self-consistency only as weak signal;
user edits, rephrases, escalations, resolutions;
targeted sentinel questions with known evidence;
periodic human audit stratified by risk/disagreement.

Proxy improvement is not proof. Maintain a rolling adjudicated set and replay it on every version. Avoid training and evaluating on the same feedback/judge.

16. Drift

Query drift

Language, products, user populations, intents, or attack patterns change. Monitor embeddings/terms/task mix and performance slices.

Corpus drift

New domains, formats, languages, authorities, duplication, and update rates can break parsers/chunkers/indexes.

Model/index drift

Provider/model updates, tokenizer changes, new embeddings, or ANN compaction shift score/rank distributions and calibration.

Evaluation drift

Dynamic facts and live web benchmarks change ground truth. Preserve snapshots and refresh a separate current set.

Trigger investigation/recalibration/reindexing by measured performance or distribution shift, not a fixed calendar alone.

17. Experiment and release workflow

Offline

register immutable corpus/query/qrels version;
run component and end-to-end baselines;
save per-query candidates/context/output/metrics/latency/cost;
compute paired deltas and slices;
run adversarial/security/deletion/freshness suites;
human-audit high-risk and disagreements;
produce signed release report.

Shadow

Run candidate version on production requests without serving output. Enforce privacy and cost limits. Compare routes, candidates, answers/judges, latency, and failures. Shadow traffic lacks user outcome for the candidate and may not cover rare cases.

Canary/gradual rollout

Route a small representative share, monitor hard gates and error budgets, then increase. Use stable assignment for comparison; exclude or analyze users whose feedback carries across variants.

Rollback

Switch atomically to prior model/prompt/index/config generation. Rollback must include caches and incompatible query/document encoders. Rehearse it.

18. Feature flags and configuration

Version every behavior-changing parameter:

source/index generation and filters;
chunker/embedding/analyzer;
retriever weights/depths/ANN parameters;
rewrite/router/policy;
reranker/selector/compressor;
context order/budget/template;
generator/prompt/decoding;
verifier/judge/threshold;
cache/fallback/timeout;
tool schemas and permissions.

Store resolved config in each trace. Avoid mutable global defaults that make two requests with the same version label behave differently.

19. Cost accounting

Per request:

C=C_{queryembed}+C_{search}+C_{rerank}+C_{prompt} +C_{decode}+C_{tools}+C_{verify}+C_{network}.

Amortized platform cost adds ingestion/parsing/enrichment/embedding/index builds, storage/replicas/backups, idle capacity, observability, evaluations, and human review.

Report cost per:

request;
answered request;
correct answer;
correct and sufficiently cited answer;
resolved user task;
tenant/domain/risk class.

Cheap incorrect answers are not efficient. Pareto-optimize quality, latency, cost, and risk; use hard gates for ACL/safety.

20. Efficiency levers

Corpus/index

better deduplication and unit design;
learned sparse or hybrid at appropriate depth;
vector dimension/precision/PQ;
HNSW/IVF/DiskANN tuning;
Matryoshka embeddings and adaptive reranking;
hot/cold tiers and selective replication;
incremental builds and reusable content hashes.

Query/retrieval

route simple queries to cheap paths;
run retrievers concurrently;
allocate candidate depths by marginal recall;
early stop on sufficiency;
cache safe repeated retrieval;
avoid duplicate query variants/results.

Context/generation

set/coverage selection;
extractive/validated compression;
smaller model for rewrite/rank/extract;
model routing;
prefix/KV caching under privacy controls;
batching and quantization;
speculative decoding/retrieval;
bounded verification targeted to risk.

Every lever requires an equal-quality or equal-cost comparison.

21. Vector/search platform selection principles

Evaluate capabilities, not marketing labels:

sparse, dense, multi-vector, hybrid and exact search;
filter expressiveness and filter-aware ANN;
update/delete/transaction semantics;
index algorithms/parameters and exact recall audit;
tenant/security isolation and authorization integration;
metadata/source storage limits;
consistency/replication/backups/disaster recovery;
throughput/tail latency at corpus/vector/filter shape;
observability and per-query explainability;
import/export/portability and lock-in;
region/compliance/encryption/key management;
cost including replicas, egress, and rebuilds;
operational maturity and failure modes.

Run a workload-specific benchmark with exact qrels, concurrency, updates, deletes, and high-selectivity filters. A vendor’s ANN benchmark without RAG evidence or tenant filters is insufficient.

22. Build versus buy

Managed platforms reduce operational burden; self-hosting can improve control, privacy, specialized algorithms, or cost at scale. Hybrid approaches use managed object/relational stores plus custom retrieval services.

Decide per capability:

commodity connectors/parsing/search may be bought;
domain chunking, policy, evaluation, routing, and quality data are often core;
high-security or specialized multimodal/graph retrieval may require custom controls;
retain exportable canonical corpus, embeddings/IDs, qrels, and traces so the system is portable.

The difficult asset is usually the governed corpus and evaluation set, not an orchestration framework.

23. Multi-region and disaster recovery

Define data residency, active-active/passive topology, index replication, source-of-truth raw store, rebuild time, recovery point/time objectives, and failover behavior. A vector index may be reproducible from canonical artifacts; test rebuild and validate hashes/counts/qrels before serving.

Cross-region replication of private documents, queries, logs, or model prompts may violate policy. Keep region-aware routing and avoid global caches for restricted data. Failover must preserve ACL/policy generation.

24. Testing pyramid

Unit/property

Chunk offsets, hashes, ACL propagation, filters, score/fusion math, token budget, source-ID validation, tombstones, cache keys.

Component

Parser fidelity, retriever qrels, ANN recall, reranker candidate set, compression preservation, generator gold-context behavior, tool schemas.

Integration

Identity -> authorized retrieval -> rerank -> context -> answer/citations; index update/delete -> cache invalidation; dependency timeout -> degraded mode.

End-to-end

Realistic questions with answerable/partial/no-answer, temporal, multi-hop, table/visual, multilingual, adversarial, and high-risk slices.

Load/chaos

Concurrency, shard/node loss, slow external API, stale replica, retry storm, large file/context, selective filters, index cutover, rollback, and region failover.

25. Failure taxonomy and runbooks

Retrieval quality incident

Check corpus/index generation, source counts, parser failures, embedding stats, ANN recall, filter/ACL changes, score distributions, router/fusion/reranker, then generator. Replay known queries on prior generation.

Freshness incident

Check connector watermarks, CDC queues, parser/index backlog, replica generation, cache invalidation, temporal metadata, and source availability. Block or label volatile answers if SLO breached.

Latency/cost incident

Locate stage and query slice; check saturation, cache hit, candidate/context/ output growth, loops/retries, filters/shard skew, external dependencies. Apply bounded degraded mode and preserve quality/safety gates.

Citation/grounding incident

Freeze affected versions, inspect evidence survival and citation mapping, validate source access/hash, generator/verifier changes, and prompt/context order. Do not merely raise a similarity threshold.

Security/privacy incident

Follow the security chapter: contain source/index/tool/tenant route, preserve authorized forensic state, rotate credentials, rebuild clean generation, invalidate caches, test targeted regression, and disclose/remediate under policy.

26. Reproducibility manifest

Every release/evaluation stores:

code commit and dependency/container hashes
corpus/query/qrels snapshots and time
source/parser/chunker/enrichment versions
embedding/analyzer/index algorithm and parameters
retriever/fusion/reranker/selector/compressor
generator/prompt/decoding/tool schemas
judge/verifier/threshold/human guidelines
hardware, concurrency, cache state
seeds, raw per-query trace, latency, tokens, cost
security/freshness/deletion tests

Model brand names without exact revisions and mutable web retrieval without saved evidence do not reproduce a run.

27. Reference deployment patterns

Curated knowledge assistant

Snapshot/CDC -> structure-aware chunks -> BM25 + exact/ANN dense -> RRF -> cross-encoder -> MMR/coverage -> cited generator -> support check -> answer.

High-security enterprise assistant

Tenant/source identity -> pre-ACL partition/filter -> local embedding/rerank/ generation as policy requires -> source trust/injection controls -> output DLP -> immutable minimal audit. Public and private indexes fuse only after authorization.

Global corpus analyst

Text hybrid retrieval plus hierarchical/community reports -> query-type route -> map-reduce/global synthesis -> claim-level retrieval to primary sources -> cited report. High ingest cost is accepted for global questions.

Live factual assistant

Task/time router -> allowlisted web/APIs and static corpus -> authority/time selection -> structured calculation -> explicit as-of answer -> short generation/ retrieval caches keyed by time/version -> stale-answer monitoring.

Visual document assistant

Native/OCR/layout/table parse plus page-image embeddings -> text/visual hybrid -> page/region rerank -> VLM/LLM with region/source IDs -> visual citation and distortion tests.

28. Production readiness checklist

governed sources, identity, license, retention, deletion lineage;
reproducible parser/chunker/embedding/index generations;
BM25/hybrid/component baselines and product gold set;
exact-versus-ANN and filtered retrieval validation;
claim citations, sufficiency, abstention, and high-risk human gate;
injection/poison/cross-tenant/privacy/DoS red teams;
p50/p95/p99 and load/chaos/degraded-mode results;
per-success quality/latency/cost Pareto analysis;
privacy-minimized end-to-end traces and dashboards;
staged rollout, rollback, incident owners/runbooks;
freshness and deletion SLOs;
periodic drift/human/evaluator audit.

29. What the executable notebooks model

The production notebook builds a release manifest, stage latency/cost trace, cache key with tenant/index/as-of scope, SLO/error-budget report, Pareto frontier, blue/green index cutover, fallback simulation, and per-query failure attribution. Load and costs are deterministic simulations so the notebook runs offline.

It does not benchmark a specific vendor or claim toy latency matches a deployed system. It teaches the measurements and invariants required to run a real one.