{ "cells": [ { "cell_type": "markdown", "metadata": { "tags": [ "field-notebook-cover", "field-notebook-style" ] }, "source": [ "\n", "\n", "
\n", "
LAB 06 · MANY FORMS
\n", "

Evidence Beyond the Paragraph

\n", "
Graphs, hierarchy, tables, page images, and domain-shaped retrieval
\n", "
not every answer lives in a rectangular chunk of text
\n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The working page\n", "\n", "“GraphRAG” and “multimodal RAG” name families, not single algorithms. This\n", "lab decomposes representations, construction, retrieval, generation, and\n", "evaluation so the extra structure is justified by the task.\n", "\n", "**Learning outcomes**\n", "\n", "- distinguish curated KGs, extracted entity graphs, passage graphs, query-time graphs, and community-report systems;\n", "- run Personalized PageRank and blend graph propagation with retrieval seeds;\n", "- select hierarchical evidence without double-counting descendant leaves;\n", "- retrieve table rows with schema/numeric signals and row provenance;\n", "- compute ColBERT/ColPali-style MaxSim and quantify vector pooling;\n", "- choose text, structure, or pixels based on the evidence—not fashion.\n", "\n", "Companion chapter: [Structured and multimodal RAG](../research/structured_and_multimodal_rag.md).\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Repository root: resolved from the notebook location\n" ] } ], "source": [ "from pathlib import Path\n", "import sys\n", "\n", "ROOT = Path.cwd()\n", "if not (ROOT / \"src\").exists():\n", " ROOT = ROOT.parent\n", "sys.path.insert(0, str(ROOT / \"src\"))\n", "print(\"Repository root: resolved from the notebook location\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Graph representations answer different questions\n", "\n", "A curated knowledge graph has typed canonical entities and relations. An\n", "OpenIE graph extracts noisy triples from text. A passage graph links chunks\n", "by entity overlap, citations, hyperlinks, or learned edges. A hierarchical\n", "graph organizes documents/sections/summaries. Microsoft GraphRAG builds an\n", "entity/relation graph, clusters communities, generates community reports,\n", "and map-reduces global questions. Query-specific systems build a small graph\n", "during search. These have different build cost, freshness, and failure modes.\n", "\n", "Use graph structure when relation chains, neighborhoods, corpus-wide themes,\n", "hierarchy, or path explanations matter. Plain hybrid retrieval often wins on\n", "local fact lookup. Graph construction cannot recover relations omitted or\n", "hallucinated by extraction.\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Personalized PageRank:\n", "entity:DPR 0.3242\n", "entity:RAG 0.2068\n", "passage:dpr 0.1837\n", "query:DPR 0.15\n", "passage:rag 0.1352\n", "Mass: 1.0\n" ] } ], "source": [ "from rag_evolution.structured import personalized_pagerank\n", "\n", "graph = {\n", " \"query:DPR\": {\"entity:DPR\": 1.0},\n", " \"entity:DPR\": {\"passage:dpr\": 1.0, \"entity:RAG\": 0.5},\n", " \"entity:RAG\": {\"passage:rag\": 1.0, \"entity:DPR\": 0.3},\n", " \"passage:dpr\": {\"entity:DPR\": 1.0},\n", " \"passage:rag\": {\"entity:RAG\": 1.0},\n", "}\n", "ranks = personalized_pagerank(graph, {\"query:DPR\": 1.0}, damping=0.85)\n", "print(\"Personalized PageRank:\")\n", "for node, score in sorted(ranks.items(), key=lambda item: -item[1]):\n", " print(node, round(score, 4))\n", "print(\"Mass:\", round(sum(ranks.values()), 8))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Graph retrieval is seed, propagate, filter, and ground\n", "\n", "Entity linking maps query mentions to graph seeds. Personalized PageRank,\n", "path search, beam search, subgraph matching, GNN scoring, or LLM-guided\n", "traversal propagates relevance. The system then maps nodes/edges back to\n", "source passages; without that last step a graph answer may be structurally\n", "plausible but ungrounded.\n", "\n", "Evaluate entity-link accuracy, edge/triple precision/recall, supporting-path\n", "recall, passage recall, answer/citation quality, build/update cost, graph\n", "storage, and performance when the graph is incomplete or conflicting.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Blended retrieval + graph propagation:\n", "entity:DPR 0.5584\n", "entity:RAG 0.3356\n", "passage:rag 0.1279\n", "passage:dpr 0.1181\n", "query:DPR 0.0\n" ] } ], "source": [ "from rag_evolution.structured import graph_expand\n", "\n", "seeds = {\"entity:DPR\": 1.0, \"entity:RAG\": 0.4}\n", "expanded = graph_expand(seeds, graph, k=6, propagation_weight=0.65)\n", "print(\"Blended retrieval + graph propagation:\")\n", "for node, score in expanded:\n", " print(node, round(score, 4))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Corpus-wide GraphRAG is not neighborhood expansion\n", "\n", "Global sensemaking systems extract entities/relations, run community\n", "detection (often Leiden), precompute hierarchical reports, select relevant\n", "communities, and aggregate partial answers. This can improve broad questions\n", "such as “What themes and actors shape this corpus?” but shifts cost to\n", "ingestion and update propagation. Dynamic community selection and DRIFT-like\n", "global-to-local refinement reduce wasted report reads.\n", "\n", "Test global synthesis and local fact questions separately. Use human factual\n", "audits in addition to LLM-judged comprehensiveness/diversity. Measure omitted\n", "facts, report hallucinations, extraction errors, update latency, and token\n", "cost. Never generalize a win on global summaries to all QA.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dynamic community selection proxy: [('control', 0.3333333333333333), ('generation', 0.3333333333333333), ('retrieval', 0.3333333333333333)]\n", "Selected reports: ['control', 'generation', 'retrieval']\n" ] } ], "source": [ "communities = {\n", " \"retrieval\": {\"DPR\", \"ColBERT\", \"SPLADE\", \"BM25\"},\n", " \"generation\": {\"RAG\", \"FiD\", \"RETRO\", \"Atlas\"},\n", " \"control\": {\"Self-RAG\", \"Adaptive-RAG\", \"Search-R1\", \"GRIP\"},\n", "}\n", "query_entities = {\"DPR\", \"RAG\", \"GRIP\"}\n", "scored = sorted(\n", " ((name, len(members & query_entities) / len(query_entities)) for name, members in communities.items()),\n", " key=lambda item: (-item[1], item[0]),\n", ")\n", "print(\"Dynamic community selection proxy:\", scored)\n", "print(\"Selected reports:\", [name for name, score in scored if score > 0])\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Hierarchical retrieval changes granularity during search\n", "\n", "RAPTOR recursively clusters and summarizes chunks into a tree. Parent–child\n", "indexes retrieve small units and expand context. Document/section trees can\n", "first route coarsely, then search leaves. Hierarchies help holistic long-\n", "document questions but summaries are lossy, can hallucinate, and must be\n", "rebuilt upward after edits. Keep leaf provenance and evaluate evidence lost\n", "in every summary level.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Selected nodes: ['dpr-leaf', 'rag-leaf', 'retrieval']\n", "Leaf evidence coverage: ('dpr', 'rag') tokens 32\n" ] } ], "source": [ "from rag_evolution.structured import HierarchyNode, select_hierarchy\n", "\n", "nodes = (\n", " HierarchyNode(\"root-summary\", \"retrieval and generation overview\", children=(\"retrieval\", \"generation\"), evidence_ids=(\"dpr\", \"rag\", \"fid\"), token_cost=40),\n", " HierarchyNode(\"retrieval\", \"dense and sparse retrieval\", evidence_ids=(\"dpr\",), token_cost=16),\n", " HierarchyNode(\"generation\", \"latent and fusion generation\", evidence_ids=(\"rag\", \"fid\"), token_cost=20),\n", " HierarchyNode(\"dpr-leaf\", \"DPR evidence\", evidence_ids=(\"dpr\",), token_cost=8),\n", " HierarchyNode(\"rag-leaf\", \"RAG evidence\", evidence_ids=(\"rag\",), token_cost=8),\n", ")\n", "selection = select_hierarchy(\n", " nodes,\n", " {\"root-summary\": 0.75, \"retrieval\": 0.8, \"generation\": 0.7, \"dpr-leaf\": 0.95, \"rag-leaf\": 0.9},\n", " token_budget=32,\n", ")\n", "print(\"Selected nodes:\", [node.identifier for node in selection.nodes])\n", "print(\"Leaf evidence coverage:\", selection.evidence_ids, \"tokens\", selection.spent)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Tables require structural and numeric semantics\n", "\n", "Flattening a table may detach headers, units, footnotes, merged cells, and row\n", "relationships. Alternatives include row/column serialization, table-aware\n", "encoders, SQL generation over governed schemas, hybrid text+cell indexes,\n", "region/image retrieval, or cell graphs. Preserve table ID, page, bounding\n", "box, row/column headers, units, and source version through citations.\n", "\n", "Test exact numeric questions, aggregations, comparisons, joins, temporal\n", "versions, missing values, unit conversion, and adversarially similar rows.\n", "Exact match on an answer is insufficient if the cited row is wrong.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "self-rag 0.825 lexical 0.833 numeric 1.0 page 4\n", "baseline 0.442 lexical 0.5 numeric 0.3333333333333333 page 4\n", "grip 0.125 lexical 0.167 numeric 0.0 page 8\n" ] } ], "source": [ "from rag_evolution.structured import TableRow, retrieve_table_rows\n", "\n", "rows = (\n", " TableRow(\"rag-results\", \"self-rag\", {\"method\": \"Self-RAG 7B\", \"PopQA\": \"54.9\", \"year\": \"2024\"}, page=4, source=\"paper-a.pdf\"),\n", " TableRow(\"rag-results\", \"baseline\", {\"method\": \"RAG baseline\", \"PopQA\": \"43.5\", \"year\": \"2024\"}, page=4, source=\"paper-a.pdf\"),\n", " TableRow(\"rag-results\", \"grip\", {\"method\": \"GRIP 8B\", \"average\": \"41.0\", \"year\": \"2026\"}, page=8, source=\"paper-b.pdf\"),\n", ")\n", "for hit in retrieve_table_rows(\"Which 2024 method reports PopQA 54.9?\", rows):\n", " print(hit.row.row_id, round(hit.score, 3), \"lexical\", round(hit.lexical_overlap, 3), \"numeric\", hit.numeric_overlap, \"page\", hit.row.page)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Visual-document RAG can bypass destructive parsing\n", "\n", "ColPali encodes rendered page patches and uses late interaction between query\n", "tokens and patch vectors. VisRAG retrieves page images and answers with a\n", "vision-language model. Visual approaches preserve layout, charts, equations,\n", "typography, and spatial relationships that text parsing can lose, but store\n", "many vectors per page and still need page/region attribution. Dynamic visual\n", "token compression reduces generation cost.\n", "\n", "Parsed text may win on clean prose and exact string search; pixels may win on\n", "tables/forms/figures. Hybrid systems can index both. Evaluate clean and\n", "degraded scans, multilingual pages, paraphrases, page retrieval, region\n", "localization, answer support, storage, latency, and token cost.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MaxSim complete page: 1.9\n", "MaxSim missing concept: 0.8\n", "pool 1 vectors 4 score 1.9\n", "pool 2 vectors 2 score 1.0\n", "pool 4 vectors 1 score 0.4\n" ] } ], "source": [ "from rag_evolution.structured import late_interaction_score, pool_vectors\n", "\n", "query_patches = ((1.0, 0.0, 0.0), (0.0, 1.0, 0.0)) # two query-token vectors\n", "page_with_table_and_title = ((0.9, 0.1, 0.0), (0.0, 1.0, 0.1), (0.1, 0.0, 0.9), (-0.5, 0.0, 0.0))\n", "text_only_page = ((0.8, 0.0, 0.1), (0.7, 0.0, 0.2))\n", "print(\"MaxSim complete page:\", round(late_interaction_score(query_patches, page_with_table_and_title), 3))\n", "print(\"MaxSim missing concept:\", round(late_interaction_score(query_patches, text_only_page), 3))\n", "for group in (1, 2, 4):\n", " pooled = pool_vectors(page_with_table_and_title, group)\n", " print(\"pool\", group, \"vectors\", len(pooled), \"score\", round(late_interaction_score(query_patches, pooled), 3))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Other modalities need modality-native evidence contracts\n", "\n", "Image RAG may retrieve global images, objects, regions, captions, OCR, or\n", "generated descriptions. Chart RAG needs axes, legends, series, marks, and\n", "visual comparison. Audio/video RAG needs transcripts, speakers, timecodes,\n", "shots, keyframes, acoustic/visual embeddings, and temporal alignment. A\n", "citation should open the exact region or time span, not merely the file.\n", "\n", "Multimodal fusion can happen at retrieval (separate indexes + fusion), in a\n", "shared embedding space, through a graph, or in the generator. Evaluate each\n", "modality alone, oracle evidence, fused evidence, missing/corrupt modalities,\n", "and cross-modal contradictions.\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "text retrieval-unit=passage citation=character span\n", "table retrieval-unit=row/cell citation=page + bounding box + headers\n", "image retrieval-unit=region citation=image + bounding box\n", "audio retrieval-unit=speaker segment citation=start/end time\n", "video retrieval-unit=shot/keyframe citation=time range + region\n", "code retrieval-unit=symbol citation=repository + commit + path + lines\n" ] } ], "source": [ "modality_units = {\n", " \"text\": (\"passage\", \"character span\"),\n", " \"table\": (\"row/cell\", \"page + bounding box + headers\"),\n", " \"image\": (\"region\", \"image + bounding box\"),\n", " \"audio\": (\"speaker segment\", \"start/end time\"),\n", " \"video\": (\"shot/keyframe\", \"time range + region\"),\n", " \"code\": (\"symbol\", \"repository + commit + path + lines\"),\n", "}\n", "for modality, (unit, citation) in modality_units.items():\n", " print(f\"{modality:6} retrieval-unit={unit:16} citation={citation}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. Code, web/API, multilingual, and regulated domains\n", "\n", "Code RAG indexes symbols, definitions, references, call/import graphs,\n", "repository paths, commits, tests, and generated artifacts; retrieval should\n", "respect repository revision. Web/API RAG needs live search provenance,\n", "robots/licenses, time snapshots, structured tool schemas, and defenses\n", "against untrusted pages. Multilingual RAG must test query/document language\n", "pairs, scripts, tokenization, transliteration, translation loss, and citation\n", "language—not just English averages.\n", "\n", "Biomedical, legal, financial, scientific, and enterprise RAG add ontology,\n", "authority, jurisdiction, valid-time, version, evidence hierarchy, access\n", "control, privacy, and calibrated abstention requirements. Domain adaptation\n", "cannot substitute for source governance or expert review.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TASK: exact error code in repository \n", " USE: lexical + symbol/call graph \n", " PROOF: commit/path/lines\n", "TASK: portfolio value on a historical date \n", " USE: table/SQL + bitemporal filter \n", " PROOF: row + snapshot\n", "TASK: themes across 50k reports \n", " USE: community reports + local verification \n", " PROOF: report claims -> leaves\n", "TASK: answer from scanned forms \n", " USE: visual page/region retrieval \n", " PROOF: page + bounding boxes\n", "TASK: simple policy definition \n", " USE: hybrid text + reranker \n", " PROOF: immutable paragraph span\n" ] } ], "source": [ "decision_examples = (\n", " (\"exact error code in repository\", \"lexical + symbol/call graph\", \"commit/path/lines\"),\n", " (\"portfolio value on a historical date\", \"table/SQL + bitemporal filter\", \"row + snapshot\"),\n", " (\"themes across 50k reports\", \"community reports + local verification\", \"report claims -> leaves\"),\n", " (\"answer from scanned forms\", \"visual page/region retrieval\", \"page + bounding boxes\"),\n", " (\"simple policy definition\", \"hybrid text + reranker\", \"immutable paragraph span\"),\n", ")\n", "for task, architecture, proof in decision_examples:\n", " print(\"TASK:\", task, \"\\n USE:\", architecture, \"\\n PROOF:\", proof)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Evaluation and selection rules\n", "\n", "Compare against strong lexical, dense, hybrid, and long-context baselines.\n", "For graphs report construction and update cost; for hierarchies report\n", "summary loss; for tables report cell/row fidelity; for visual systems report\n", "storage and region attribution; for every system report end-to-end answer and\n", "citation support. Use task slices rather than a single average.\n", "\n", "**This lab does not claim graphs or pixels are universally superior.** It\n", "provides the algorithms and measurements that expose when relational,\n", "hierarchical, spatial, numeric, temporal, or multimodal signal earns its\n", "additional complexity.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.9" }, "rag_evolution": { "presentation": "plain-field-notebook", "visual_version": 2 } }, "nbformat": 4, "nbformat_minor": 5 }