Evidence Beyond the Paragraph
The working page
“GraphRAG” and “multimodal RAG” name families, not single algorithms. This lab decomposes representations, construction, retrieval, generation, and evaluation so the extra structure is justified by the task.
Learning outcomes
- distinguish curated KGs, extracted entity graphs, passage graphs, query-time graphs, and community-report systems;
- run Personalized PageRank and blend graph propagation with retrieval seeds;
- select hierarchical evidence without double-counting descendant leaves;
- retrieve table rows with schema/numeric signals and row provenance;
- compute ColBERT/ColPali-style MaxSim and quantify vector pooling;
- choose text, structure, or pixels based on the evidence—not fashion.
Companion chapter: Structured and multimodal RAG.
from pathlib import Path
import sys
ROOT = Path.cwd()
if not (ROOT / "src").exists():
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / "src"))
print("Repository root: resolved from the notebook location")
Repository root: resolved from the notebook location
1. Graph representations answer different questions
A curated knowledge graph has typed canonical entities and relations. An OpenIE graph extracts noisy triples from text. A passage graph links chunks by entity overlap, citations, hyperlinks, or learned edges. A hierarchical graph organizes documents/sections/summaries. Microsoft GraphRAG builds an entity/relation graph, clusters communities, generates community reports, and map-reduces global questions. Query-specific systems build a small graph during search. These have different build cost, freshness, and failure modes.
Use graph structure when relation chains, neighborhoods, corpus-wide themes, hierarchy, or path explanations matter. Plain hybrid retrieval often wins on local fact lookup. Graph construction cannot recover relations omitted or hallucinated by extraction.
from rag_evolution.structured import personalized_pagerank
graph = {
"query:DPR": {"entity:DPR": 1.0},
"entity:DPR": {"passage:dpr": 1.0, "entity:RAG": 0.5},
"entity:RAG": {"passage:rag": 1.0, "entity:DPR": 0.3},
"passage:dpr": {"entity:DPR": 1.0},
"passage:rag": {"entity:RAG": 1.0},
}
ranks = personalized_pagerank(graph, {"query:DPR": 1.0}, damping=0.85)
print("Personalized PageRank:")
for node, score in sorted(ranks.items(), key=lambda item: -item[1]):
print(node, round(score, 4))
print("Mass:", round(sum(ranks.values()), 8))
Personalized PageRank: entity:DPR 0.3242 entity:RAG 0.2068 passage:dpr 0.1837 query:DPR 0.15 passage:rag 0.1352 Mass: 1.0
2. Graph retrieval is seed, propagate, filter, and ground
Entity linking maps query mentions to graph seeds. Personalized PageRank, path search, beam search, subgraph matching, GNN scoring, or LLM-guided traversal propagates relevance. The system then maps nodes/edges back to source passages; without that last step a graph answer may be structurally plausible but ungrounded.
Evaluate entity-link accuracy, edge/triple precision/recall, supporting-path recall, passage recall, answer/citation quality, build/update cost, graph storage, and performance when the graph is incomplete or conflicting.
from rag_evolution.structured import graph_expand
seeds = {"entity:DPR": 1.0, "entity:RAG": 0.4}
expanded = graph_expand(seeds, graph, k=6, propagation_weight=0.65)
print("Blended retrieval + graph propagation:")
for node, score in expanded:
print(node, round(score, 4))
Blended retrieval + graph propagation: entity:DPR 0.5584 entity:RAG 0.3356 passage:rag 0.1279 passage:dpr 0.1181 query:DPR 0.0
3. Corpus-wide GraphRAG is not neighborhood expansion
Global sensemaking systems extract entities/relations, run community detection (often Leiden), precompute hierarchical reports, select relevant communities, and aggregate partial answers. This can improve broad questions such as “What themes and actors shape this corpus?” but shifts cost to ingestion and update propagation. Dynamic community selection and DRIFT-like global-to-local refinement reduce wasted report reads.
Test global synthesis and local fact questions separately. Use human factual audits in addition to LLM-judged comprehensiveness/diversity. Measure omitted facts, report hallucinations, extraction errors, update latency, and token cost. Never generalize a win on global summaries to all QA.
communities = {
"retrieval": {"DPR", "ColBERT", "SPLADE", "BM25"},
"generation": {"RAG", "FiD", "RETRO", "Atlas"},
"control": {"Self-RAG", "Adaptive-RAG", "Search-R1", "GRIP"},
}
query_entities = {"DPR", "RAG", "GRIP"}
scored = sorted(
((name, len(members & query_entities) / len(query_entities)) for name, members in communities.items()),
key=lambda item: (-item[1], item[0]),
)
print("Dynamic community selection proxy:", scored)
print("Selected reports:", [name for name, score in scored if score > 0])
Dynamic community selection proxy: [('control', 0.3333333333333333), ('generation', 0.3333333333333333), ('retrieval', 0.3333333333333333)]
Selected reports: ['control', 'generation', 'retrieval']
4. Hierarchical retrieval changes granularity during search
RAPTOR recursively clusters and summarizes chunks into a tree. Parent–child indexes retrieve small units and expand context. Document/section trees can first route coarsely, then search leaves. Hierarchies help holistic long- document questions but summaries are lossy, can hallucinate, and must be rebuilt upward after edits. Keep leaf provenance and evaluate evidence lost in every summary level.
from rag_evolution.structured import HierarchyNode, select_hierarchy
nodes = (
HierarchyNode("root-summary", "retrieval and generation overview", children=("retrieval", "generation"), evidence_ids=("dpr", "rag", "fid"), token_cost=40),
HierarchyNode("retrieval", "dense and sparse retrieval", evidence_ids=("dpr",), token_cost=16),
HierarchyNode("generation", "latent and fusion generation", evidence_ids=("rag", "fid"), token_cost=20),
HierarchyNode("dpr-leaf", "DPR evidence", evidence_ids=("dpr",), token_cost=8),
HierarchyNode("rag-leaf", "RAG evidence", evidence_ids=("rag",), token_cost=8),
)
selection = select_hierarchy(
nodes,
{"root-summary": 0.75, "retrieval": 0.8, "generation": 0.7, "dpr-leaf": 0.95, "rag-leaf": 0.9},
token_budget=32,
)
print("Selected nodes:", [node.identifier for node in selection.nodes])
print("Leaf evidence coverage:", selection.evidence_ids, "tokens", selection.spent)
Selected nodes: ['dpr-leaf', 'rag-leaf', 'retrieval']
Leaf evidence coverage: ('dpr', 'rag') tokens 32
5. Tables require structural and numeric semantics
Flattening a table may detach headers, units, footnotes, merged cells, and row relationships. Alternatives include row/column serialization, table-aware encoders, SQL generation over governed schemas, hybrid text+cell indexes, region/image retrieval, or cell graphs. Preserve table ID, page, bounding box, row/column headers, units, and source version through citations.
Test exact numeric questions, aggregations, comparisons, joins, temporal versions, missing values, unit conversion, and adversarially similar rows. Exact match on an answer is insufficient if the cited row is wrong.
from rag_evolution.structured import TableRow, retrieve_table_rows
rows = (
TableRow("rag-results", "self-rag", {"method": "Self-RAG 7B", "PopQA": "54.9", "year": "2024"}, page=4, source="paper-a.pdf"),
TableRow("rag-results", "baseline", {"method": "RAG baseline", "PopQA": "43.5", "year": "2024"}, page=4, source="paper-a.pdf"),
TableRow("rag-results", "grip", {"method": "GRIP 8B", "average": "41.0", "year": "2026"}, page=8, source="paper-b.pdf"),
)
for hit in retrieve_table_rows("Which 2024 method reports PopQA 54.9?", rows):
print(hit.row.row_id, round(hit.score, 3), "lexical", round(hit.lexical_overlap, 3), "numeric", hit.numeric_overlap, "page", hit.row.page)
self-rag 0.825 lexical 0.833 numeric 1.0 page 4 baseline 0.442 lexical 0.5 numeric 0.3333333333333333 page 4 grip 0.125 lexical 0.167 numeric 0.0 page 8
6. Visual-document RAG can bypass destructive parsing
ColPali encodes rendered page patches and uses late interaction between query tokens and patch vectors. VisRAG retrieves page images and answers with a vision-language model. Visual approaches preserve layout, charts, equations, typography, and spatial relationships that text parsing can lose, but store many vectors per page and still need page/region attribution. Dynamic visual token compression reduces generation cost.
Parsed text may win on clean prose and exact string search; pixels may win on tables/forms/figures. Hybrid systems can index both. Evaluate clean and degraded scans, multilingual pages, paraphrases, page retrieval, region localization, answer support, storage, latency, and token cost.
from rag_evolution.structured import late_interaction_score, pool_vectors
query_patches = ((1.0, 0.0, 0.0), (0.0, 1.0, 0.0)) # two query-token vectors
page_with_table_and_title = ((0.9, 0.1, 0.0), (0.0, 1.0, 0.1), (0.1, 0.0, 0.9), (-0.5, 0.0, 0.0))
text_only_page = ((0.8, 0.0, 0.1), (0.7, 0.0, 0.2))
print("MaxSim complete page:", round(late_interaction_score(query_patches, page_with_table_and_title), 3))
print("MaxSim missing concept:", round(late_interaction_score(query_patches, text_only_page), 3))
for group in (1, 2, 4):
pooled = pool_vectors(page_with_table_and_title, group)
print("pool", group, "vectors", len(pooled), "score", round(late_interaction_score(query_patches, pooled), 3))
MaxSim complete page: 1.9 MaxSim missing concept: 0.8 pool 1 vectors 4 score 1.9 pool 2 vectors 2 score 1.0 pool 4 vectors 1 score 0.4
7. Other modalities need modality-native evidence contracts
Image RAG may retrieve global images, objects, regions, captions, OCR, or generated descriptions. Chart RAG needs axes, legends, series, marks, and visual comparison. Audio/video RAG needs transcripts, speakers, timecodes, shots, keyframes, acoustic/visual embeddings, and temporal alignment. A citation should open the exact region or time span, not merely the file.
Multimodal fusion can happen at retrieval (separate indexes + fusion), in a shared embedding space, through a graph, or in the generator. Evaluate each modality alone, oracle evidence, fused evidence, missing/corrupt modalities, and cross-modal contradictions.
modality_units = {
"text": ("passage", "character span"),
"table": ("row/cell", "page + bounding box + headers"),
"image": ("region", "image + bounding box"),
"audio": ("speaker segment", "start/end time"),
"video": ("shot/keyframe", "time range + region"),
"code": ("symbol", "repository + commit + path + lines"),
}
for modality, (unit, citation) in modality_units.items():
print(f"{modality:6} retrieval-unit={unit:16} citation={citation}")
text retrieval-unit=passage citation=character span table retrieval-unit=row/cell citation=page + bounding box + headers image retrieval-unit=region citation=image + bounding box audio retrieval-unit=speaker segment citation=start/end time video retrieval-unit=shot/keyframe citation=time range + region code retrieval-unit=symbol citation=repository + commit + path + lines
8. Code, web/API, multilingual, and regulated domains
Code RAG indexes symbols, definitions, references, call/import graphs, repository paths, commits, tests, and generated artifacts; retrieval should respect repository revision. Web/API RAG needs live search provenance, robots/licenses, time snapshots, structured tool schemas, and defenses against untrusted pages. Multilingual RAG must test query/document language pairs, scripts, tokenization, transliteration, translation loss, and citation language—not just English averages.
Biomedical, legal, financial, scientific, and enterprise RAG add ontology, authority, jurisdiction, valid-time, version, evidence hierarchy, access control, privacy, and calibrated abstention requirements. Domain adaptation cannot substitute for source governance or expert review.
decision_examples = (
("exact error code in repository", "lexical + symbol/call graph", "commit/path/lines"),
("portfolio value on a historical date", "table/SQL + bitemporal filter", "row + snapshot"),
("themes across 50k reports", "community reports + local verification", "report claims -> leaves"),
("answer from scanned forms", "visual page/region retrieval", "page + bounding boxes"),
("simple policy definition", "hybrid text + reranker", "immutable paragraph span"),
)
for task, architecture, proof in decision_examples:
print("TASK:", task, "\n USE:", architecture, "\n PROOF:", proof)
TASK: exact error code in repository USE: lexical + symbol/call graph PROOF: commit/path/lines TASK: portfolio value on a historical date USE: table/SQL + bitemporal filter PROOF: row + snapshot TASK: themes across 50k reports USE: community reports + local verification PROOF: report claims -> leaves TASK: answer from scanned forms USE: visual page/region retrieval PROOF: page + bounding boxes TASK: simple policy definition USE: hybrid text + reranker PROOF: immutable paragraph span
9. Evaluation and selection rules
Compare against strong lexical, dense, hybrid, and long-context baselines. For graphs report construction and update cost; for hierarchies report summary loss; for tables report cell/row fidelity; for visual systems report storage and region attribution; for every system report end-to-end answer and citation support. Use task slices rather than a single average.
This lab does not claim graphs or pixels are universally superior. It provides the algorithms and measurements that expose when relational, hierarchical, spatial, numeric, temporal, or multimodal signal earns its additional complexity.