# Graph, hierarchical, structured, multimodal, and domain-specific RAG

Vector retrieval over text passages is only one knowledge-access model. Many
questions depend on relations, hierarchy, tables, visual layout, media, code,
live APIs, or domain authority. This chapter separates technique families that
are often collapsed into the vague labels “GraphRAG” or “multimodal RAG.”

## 1. A representation-first decision

Choose the representation that preserves the operations required by the task:

| Required operation | Natural representation |
|---|---|
| exact phrase/identifier | lexical postings |
| paraphrase lookup | dense/sparse semantic vectors |
| token/region correspondence | multi-vector late interaction |
| entity relation/path | knowledge/document graph |
| hierarchy/global themes | tree/community summaries |
| filtering/aggregation/join | relational table/SQL |
| visual layout/chart | page/region image plus structure |
| code dependency/symbol use | AST, symbol and call/import graph |
| evolving event state | bitemporal event/record store |
| fresh external fact/calculation | typed API/tool |

Multiple coordinated representations are usually better than flattening
everything into text. They must share stable source IDs so results can be fused,
deduplicated, cited, and deleted.

## 2. Graph RAG is not one algorithm

A graph RAG system is defined by:

1. node types;
2. edge types and direction;
3. construction/extraction method;
4. entity resolution;
5. indexing and update policy;
6. query-to-graph linking;
7. traversal/scoring algorithm;
8. mapping from graph result to source evidence;
9. generator integration;
10. evaluation at each stage.

Common graph types:

- curated entity/relation knowledge graph;
- LLM/OpenIE-extracted corpus graph;
- passage-entity bipartite graph;
- document citation/hyperlink graph;
- section/tree hierarchy;
- event/temporal graph;
- table/schema/foreign-key graph;
- code symbol/call/dependency graph;
- conversation/entity memory graph;
- query-specific graph built during reasoning;
- similarity kNN graph over passages or embeddings.

Results from one type do not establish the value of another.

## 3. Curated knowledge-graph question answering

### Entity linking

Map query mentions to candidate entity IDs using lexical aliases, embeddings,
type/context, and popularity priors. Preserve multiple candidates when
ambiguous. A wrong seed entity creates a hard graph-recall ceiling.

### Neighborhood and path retrieval

From seed nodes, expand typed edges, score paths, or execute a symbolic query.
A path \(p=(e_0,r_1,e_1,\ldots,r_h,e_h)\) can be scored by

\[
s(p,q)=s_{seed}(e_0,q)+\sum_{i=1}^{h}s_r(r_i,q)
-\lambda h+\gamma s_{target}(e_h,q).
\]

Beam search controls branching but can discard the only useful early low-score
edge. Type, time, and direction constraints reduce the space. Return the source
provenance for each triple; a KG edge without origin/version is hard to audit.

### GraftNet, PullNet, and graph neural QA

GraftNet builds a heterogeneous graph of KG entities and text and propagates
representations. PullNet learns to iteratively retrieve nodes/relations/text
from a large knowledge base. These predate the modern GraphRAG label and
establish the retrieve-expand-reason pattern. Their answer spaces and
supervision often differ from free-form generation.

### KG-FiD

KG-FiD links entities across retrieved passages, uses graph structure to rerank
or select passages, then applies FiD. It illustrates a useful hybrid: text
retrieval supplies candidate evidence, graph relations improve multi-hop
organization, and the generator still reads source text.

## 4. Passage/entity associative graphs

[HippoRAG](https://proceedings.neurips.cc/paper_files/paper/2024/hash/6ddc81d76dc3e20c1cdbda4a040d11ae-Abstract-Conference.html)
uses OpenIE to construct an entity graph, links query entities to seeds, and
runs Personalized PageRank over connected knowledge/passages. For transition
matrix \(P\), restart distribution \(v\), and damping \(\alpha\):

\[
\pi=(1-\alpha)v+\alpha P^\top\pi.
\]

This spreads relevance through shared entities and can recover multi-hop
associations without repeated LLM queries. Entity extraction/linking errors,
popular hubs, and graph build/update costs are central.

HippoRAG 2 adds passage nodes, contextual edges, and online query integration/
filtering. Comparing it to vector RAG must account for powerful builder, reader,
and embedding models, not graph structure alone.

## 5. Microsoft GraphRAG: global corpus sensemaking

[GraphRAG](https://www.microsoft.com/en-us/research/publication/from-local-to-global-a-graph-rag-approach-to-query-focused-summarization/)
uses an LLM to extract entities/relations/claims, builds a graph, detects Leiden
communities, generates community reports, and answers global questions through
map-reduce over reports. Local search can combine entities, relations, claims,
community reports, and source text.

Its strength is global questions such as themes, actors, and relationships over
a corpus where local top-k passages are insufficient. It is not evidence that
graphs dominate simple factoid retrieval. Costs include repeated LLM extraction
and summarization, graph storage, community recomputation, update propagation,
and summary validation.

Dynamic community selection reduces report/token cost by choosing relevant
communities at query time. DRIFT seeds local iterative search from global
community information. These are distinct retrieval modes and should be
evaluated on local and global questions separately.

### Community summary provenance

Store report-to-community, community-to-node, node/edge-to-source, and exact
source spans. A claim in a community report is generated and may not be
entailed by any one source. Answers should cite original evidence and label
corpus-level inferences.

## 6. Query-specific and dynamic graphs

Corpus-wide graphs are expensive and can freeze extraction errors. Query-
specific systems retrieve text and construct only a relevant incremental graph.
[RAS](https://iclr.cc/virtual/2026/poster/10008199) interleaves targeted
retrieval with query-specific KG construction. Other systems search proposition
paths or generate retrieval programs.

Advantages:

- lower irrelevant graph volume;
- structure tailored to the current question;
- easier updates from fresh retrieval;
- can combine web/text/API evidence.

Risks:

- repeated per-query extraction latency;
- unstable graphs and plans;
- early retrieval errors shape all later structure;
- duplicated extraction across queries;
- harder caching and reproducibility.

Persist the temporary graph and source lineage in the trace.

## 7. Graph retrieval algorithms

### Seed-and-expand

Retrieve seed passages/entities lexically or densely, then take one or more
neighbors with relation/type constraints. Cheap and interpretable; fixed hop
count can under/over-expand.

### Personalized PageRank/random walk

Diffuses seed probability through graph topology. Edge weights, hub correction,
restart probability, and passage aggregation strongly affect results.

### Shortest path and k-shortest paths

Useful for explicit relational connections when edge costs are meaningful.
The shortest path can be semantically wrong or based on low-confidence edges.

### Beam search

Maintain the top \(B\) partial paths under an LLM/embedding/learned score. More
flexible than shortest path but expensive and prone to locally plausible dead
ends. Record pruned paths for diagnosis.

### Community retrieval

Rank graph communities or generated reports, possibly at several hierarchy
levels. Good for global synthesis; lossy for exact facts.

### Graph neural networks

Message-passing networks learn representations over a retrieved subgraph. They
can combine structure and text but require training graphs/labels and can
oversmooth or propagate noisy edges.

### LLM traversal/tool use

An LLM chooses entities, relations, queries, or paths. It can interpret schemas
but adds sequential cost, nondeterminism, and hallucinated relation/tool calls.
Validate every operation against the graph schema and hard budget.

## 8. Graph construction and extraction

### Open information extraction

Extract `(subject, relation, object)` triples from text. Open relation phrases
preserve nuance but fragment equivalent relations. Coreference and implicit
arguments are difficult. Align each triple to exact spans.

### Schema-guided extraction

Extract into governed entity/relation types. This improves consistency and
queryability but misses facts outside the schema and requires evolution.

### LLM extraction

LLMs can extract richer entities/relations/claims and descriptions. Use
constrained schemas, deterministic settings when possible, untrusted-data
delimiters, span verification, confidence, and model/prompt version. Never
accept generated triples without source alignment.

### Entity resolution

Normalize aliases, detect same/different entities, and preserve uncertainty.
Over-merging creates false paths; under-merging breaks connectivity. Evaluate
pairwise/coreference accuracy and downstream path recall.

### Edge confidence and contradiction

Store extraction confidence, source authority, time, and support count. Do not
collapse conflicting values into one edge. Model claims/events with validity and
provenance so temporal queries can select correctly.

## 9. Evaluating graph RAG

Separate:

1. entity/relation extraction precision/recall;
2. entity-linking accuracy;
3. graph completeness and duplicate/hub structure;
4. seed recall;
5. path/subgraph retrieval;
6. source-text retrieval after graph navigation;
7. answer/citation quality;
8. build/update/query cost;
9. deletion and provenance correctness.

[When to Use Graphs in RAG](https://iclr.cc/virtual/2026/poster/10007992)
and later modular analyses report that graph RAG frequently loses to vanilla
RAG in settings without useful relational/global structure. Compare against a
strong hybrid text baseline using the same generator and evidence budget.

## 10. Hierarchical RAG

### Structural hierarchy

Use the source’s document/section/subsection/paragraph tree. Retrieve leaves or
headings, then expand ancestors/children. This is cheap if parsing is reliable
and preserves author structure.

### Cluster hierarchy

Cluster embeddings, summarize clusters, and recurse as in RAPTOR. This creates
a latent topic tree independent of original headings. Clustering parameters and
summary prompts define the hierarchy.

### Parent-child retrieval

Index precise children and return larger parents. Deduplicate parents and merge
neighbor children. Strong for long documents without generated summaries.

### Multi-resolution index

Index document, section, passage, sentence/proposition, and summary views.
Route or fuse by query type. Global questions search coarse units; exact facts
search fine units. Cross-view identity prevents duplicates.

### Map-reduce synthesis

Map a question over chunks/communities, generate partial answers or claim sets,
then reduce. It scales global synthesis but partial generators can omit facts,
and reducer context is another bottleneck. Preserve map outputs and citations;
measure coverage as corpus/sample size increases.

## 11. Table RAG

Text flattening is weak for tables because it loses axes and types. A table RAG
pipeline includes:

1. table detection/extraction;
2. header hierarchy and cell-span reconstruction;
3. table/schema summaries and embeddings;
4. table-level candidate retrieval;
5. row/column/cell retrieval or SQL planning;
6. joins across tables/text;
7. executable aggregation/calculation;
8. cell-level provenance and answer rendering.

### Retrieval units

- whole table: global but too large;
- schema/header: good for table selection;
- row with repeated headers: fact lookup;
- column profile: distribution/attribute queries;
- cell neighborhood: exact values;
- table graph: multirow/multitable relations;
- generated summary: semantic lookup but lossy.

### SQL/program execution

Retrieve relevant tables/schema, generate a typed query, validate it, execute in
a read-only sandbox, and return result plus rows/columns used. Prefer execution
for arithmetic, filtering, joins, and aggregation. Validate types, units, NULL,
dates, and row counts. The generator should not reproduce calculations from
memory.

[TableRAG](https://aclanthology.org/2025.emnlp-main.710/) combines query
decomposition, text retrieval, SQL, and intermediate answers for heterogeneous
documents. [T-RAG](https://aclanthology.org/2026.findings-acl.1902/) uses a
hierarchical memory index and graph-aware organization over large table
corpora. These solve different table-corpus tasks; compare datasets, table
counts, and reasoning operations.

### Table evaluation

Measure table/row/cell recall, schema linking, executable-query accuracy,
denotation, numerical reasoning, provenance, and final answer. Include merged
headers, units, footnotes, multiple tables, text-table joins, and no-answer.

## 12. Visual document RAG

### Text-first pipeline

OCR/layout/table parsing -> text/structure units -> text retrieval -> LLM. It is
cheap and auditable when parsing is good. It fails on layout, figures, charts,
handwriting, and parser errors.

### Caption/description pipeline

Generate textual descriptions of pages/images/regions and index them. It
bridges to text infrastructure but descriptions omit details and can hallucinate.
Keep original visual evidence and generation lineage.

### Visual embedding pipeline

Render pages/images, encode them with a vision-language model, and retrieve by
text/image query. [VisRAG](https://proceedings.iclr.cc/paper_files/paper/2025/hash/3640e20b253c7530cce06abdd3c2361b-Abstract-Conference.html)
uses direct visual retrieval and VLM generation. Comparisons must ensure parsed-
text baselines are not artificially weak on layout-heavy tasks.

### Visual late interaction

[ColPali](https://proceedings.iclr.cc/paper_files/paper/2025/hash/99e9cf99cc114c46c2e6168e4dc0c43a-Abstract-Conference.html)
stores patch embeddings and uses MaxSim with query tokens. It reported strong
ViDoRe retrieval and fast page indexing relative to parsing, at much higher
bytes/page than single-vector text embeddings. Page retrieval does not itself
localize support or guarantee generation.

### End-to-end visual document RAG

VDocRAG adds visual retrieval and dynamic visual-token compression. MoLoRAG
uses a page graph and lightweight VLM traversal. RobustVisRAG explicitly
addresses synthetic and real distortions. MegaRAG combines textual, visual, and
spatial cues in a multimodal KG.

### Coordinated text+visual retrieval

Search OCR/text, table, and page-image indexes; fuse by page/source identity;
use a modality router/reranker; send selected text and regions to an LLM/VLM.
This provides fallback and provenance but costs more. Evaluate clean versus
distorted pages and which modality supplied each answer claim.

## 13. Images, charts, maps, audio, and video

### Images

Index global and region embeddings, OCR, captions, detected entities, and source
metadata. Region retrieval is needed for small objects/text. Generated captions
are not ground truth.

### Charts

Combine visual chart retrieval, title/axis/legend OCR, chart-to-table extraction,
and executable analysis. Questions about trends or comparisons need values and
axis semantics, not a generic caption.

### Maps and spatial evidence

Preserve coordinate systems, layers, scale, time, and region geometry. Spatial
queries should use geospatial indexes/operations; text similarity is not a
substitute for containment, intersection, or distance.

### Audio

Use transcript, acoustic/music embeddings, speaker and time segments. A text
question may require a spoken phrase; an acoustic question may require sound
events absent from transcript.

### Video

Retrieve hierarchically: video -> scene -> clip -> frame/transcript segment.
Temporal relations require ordered segments. Dense sampling is expensive;
shot/topic detection and query-conditioned frame selection trade recall and
cost.

## 14. Code and software-engineering RAG

### Knowledge units

- repository/file/module;
- class/function/method/symbol;
- AST subtree;
- docstring/comment;
- test and failure trace;
- issue/PR/commit/diff;
- dependency/configuration/API documentation.

### Retrieval signals

Lexical search is essential for identifiers and error strings. Dense code/text
embeddings bridge descriptions and code. Symbol, import, call, type, ownership,
and test graphs supply structure. Query expansion can include stack traces and
referenced symbols.

### Iterative repository retrieval

RepoCoder-style systems retrieve code, generate, then use generated context for
another retrieval iteration. Repoformer learns when retrieval is useful to
avoid distracting context. Tool-using coding agents search, inspect, edit, run
tests, and read failures; this is RAG embedded in a software control loop.

### Correctness and provenance

Use exact repository commit, language/toolchain, dependency versions, and
build/test environment. Execute tests/static analysis in a sandbox. Cite files/
symbols/lines and distinguish copied/adapted code licenses. A plausible snippet
from a different version is a retrieval failure.

## 15. Web and live-tool RAG

### Search engine retrieval

Web search provides freshness and breadth but unstable ranking, SEO/spam,
paywalls, snippets without context, and no fixed corpus. Store query, time,
engine/API version, result list, fetched pages, and hashes. Rerank by relevance,
authority, freshness, and independence.

### API/tool retrieval

Use structured APIs for current prices, weather, calculations, databases, and
system state. Validate tool schema and arguments, enforce authentication and
rate limits, distinguish missing/timeout/error from empty result, and cite the
observed response/time.

### Tool selection

A router chooses static corpus, web, KG, SQL, calculator, code execution, or
long context. Train/evaluate both tool choice and argument correctness. Hard
allowlists, scopes, and cost limits remain outside the LLM.

## 16. Multilingual and cross-lingual RAG

Architectures:

1. one multilingual embedding/sparse index;
2. per-language indexes and language router;
3. translate query into corpus language;
4. cross-lingual dense retrieval and answer in user language;
5. search multiple languages and fuse sources;
6. machine-translate documents at ingestion as an additional view.

Trade-offs include tokenizer/morphology, transliteration, named entities,
translation errors, uneven corpus authority, index size, and generator language
control. Preserve original text and cite it; translated evidence is a derived
view. Evaluate retrieval and generation per language/script, not macro averages
only. [NoMIRACL](https://aclanthology.org/2024.findings-emnlp.730/) measures
hallucination and miss across 18 languages but not full answer correctness.

Learned sparse vocabulary may transfer poorly to unseen scripts; multilingual
dense models may compress low-resource distinctions. Hybrid language-specific
analyzers remain important.

## 17. Biomedical and clinical RAG

Requirements:

- controlled terminology, synonyms, abbreviations, gene/drug identifiers;
- publication, guideline, evidence-grade, cohort, and retraction metadata;
- patient/tenant isolation and minimum necessary access;
- temporal validity and versioned clinical guidance;
- calibrated abstention and clinician review;
- distinction between literature evidence and patient-specific record;
- no unsupported diagnosis/treatment synthesis.

Retrieve from an allowlisted hierarchy: current official guidelines, systematic
reviews, primary studies, drug labels, and patient record as policy permits.
Semantic relevance does not encode evidence grade. Evaluate retrieval authority,
support, contraindications, rare cases, and harmful omission.

## 18. Legal and regulatory RAG

Preserve jurisdiction, court/agency, authority level, decision/status,
effective/amendment/repeal dates, citations, sections, and quoted text. Build a
citation graph but retain official source spans. Retrieve statutes, regulations,
cases, guidance, and contracts under different policies.

Historical/as-of questions require bitemporal versions. A later amendment is not
evidence of the earlier rule. Generators should distinguish binding authority,
persuasive authority, commentary, and user documents. Require local citations
and human legal review.

## 19. Financial RAG

Index filings, notes, tables, earnings calls, presentations, market data, and
policies with entity/security, reporting period, currency, units, filing/event/
restatement times, and source authority. Tables and calculations are central.

Use APIs/SQL for current structured values, execute ratios/aggregations, cite
inputs, and label derived calculations. Separate reported, adjusted, and analyst
figures. Avoid mixing periods/currencies or using publication time as event time.

## 20. Scientific and scholarly RAG

Represent sections, references, equations, tables, figures, methods, datasets,
and supplements. Citation graphs help discovery but popularity is not truth.
Track version/preprint/published/correction/retraction. Use claim-evidence
alignment to primary papers rather than citing a survey for an exact result.

Long-form literature synthesis needs query decomposition, diverse source
selection, study-design metadata, contradiction handling, and transparent
inference. Generated reviews should distinguish reported findings from synthesis.

## 21. Enterprise support and personal memory

Support RAG combines product documentation, versions, incidents, tickets,
runbooks, and account state. Route by product/version/platform; use current
known-issue and service-state sources; never expose another customer’s ticket.

Personal memory stores preferences, commitments, and episodes. Require consent,
write/edit/delete UI, temporal updates, source turn, sensitivity, and forgetting.
A retrieved old preference should not override a new correction. The memory
chapter covers lifecycle policies.

## 22. Cost model across representations

For corpus of \(N\) source documents producing \(U\) units, average vectors
\(v\) per unit, dimension \(d\), numeric bytes \(b\), replication \(r\):

\[
\text{raw vector bytes}\approx Uvdb r.
\]

Add graph edges, postings, PQ codes, metadata, summaries, and source images.
Graph/visual/multi-vector systems move substantial cost to ingestion and storage.
Per-query cost includes routing, retrievals, graph traversals, reranking, VLM/
LLM tokens, tools, and verification.

Compare methods at equal quality or equal cost/latency. A graph that improves
one global task but multiplies ingest cost may still be correct for that product;
it is not a universal default.

## 23. Unified evaluation matrix

| Representation | Component metrics | End-to-end stress |
|---|---|---|
| Graph | extraction/link/path/subgraph recall | multi-hop, global, updates, poisoned edges |
| Hierarchy | level/descendant evidence recall | local vs global questions, summary loss |
| Table | table/row/cell/schema/SQL accuracy | joins, aggregation, units, text-table |
| Visual document | page/region nDCG/recall | layout, tables, clean/distorted OCR, citations |
| Image/chart | image/region/data extraction | visual relations, numerical trends |
| Audio/video | segment/frame/time recall | cross-modal and temporal questions |
| Code | symbol/file/graph recall | exact revision, build/test correctness |
| Web/API | source/result/tool accuracy, freshness | dynamic replay, authority, failure/timeouts |
| Multilingual | retrieval by language/script | answer language, cross-lingual support |
| Domain | relevance + authority/time/privacy | high-risk failure and human review |

Always include a strong hybrid text baseline, oracle structured evidence, and
the same generator/context budget where possible.

## 24. Selection guide

- Use graphs when the task truly depends on relations, paths, global communities,
  or reusable entity structure.
- Use hierarchy when documents/corpora have meaningful multiresolution
  information needs.
- Use SQL/table-native operations for exact filtering, joins, and aggregation.
- Use visual retrieval when layout or visual content is evidence, not merely
  decoration.
- Use modality ensembles when parser reliability varies and cost permits.
- Use code structure and execution for software tasks.
- Use live APIs for volatile structured facts.
- Add domain authority, temporal, privacy, and review controls before model
  sophistication.

## 25. What the executable notebooks model

The structured notebook builds a typed passage/entity graph, runs seed expansion
and Personalized PageRank, compares graph and text retrieval, executes a small
table query with cell provenance, and fuses text/table/visual-proxy result lists
by source identity. It also demonstrates hierarchy, versioned graph edges, and
modality-aware citations.

These transparent objects do not reproduce LLM graph extraction, VLM page
embeddings, or production SQL security. They make representation, traversal,
lineage, cost, and evaluation choices explicit.