{ "cells": [ { "cell_type": "markdown", "metadata": { "tags": [ "field-notebook-cover", "field-notebook-style" ] }, "source": [ "\n", "\n", "
\n", "
LAB 01 · FOUNDATIONS
\n", "

From Words to Vectors

\n", "
A working notebook on sparse, semantic, and hybrid retrieval
\n", "
first, learn what the catalogue remembers
\n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The working page\n", "\n", "This notebook makes the architectural history executable. It starts with BM25, adds an independently encoded semantic representation, fuses heterogeneous rankings, and connects those components to DPR, RAG, FiD, RETRO, and modern hybrid systems.\n", "\n", "The implementation is intentionally offline and inspectable. `BM25Retriever` implements real Okapi BM25. `HashingSemanticRetriever` is a fixed-width synonym and character-ngram proxy for a dual encoder; it demonstrates the interface but is **not** a pretrained neural retriever or a paper reproduction.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Historical map\n", "\n", "| Period | Core question | Representative answer |\n", "|---|---|---|\n", "| 1972–1995 | How should exact terms be weighted? | TF-IDF, relevance weighting, BM25 |\n", "| 2014–2017 | Can raw facts be external memory? | Memory Networks, DrQA |\n", "| 2019–2020 | Can retrieval be learned from QA/LM objectives? | ORQA, REALM, DPR |\n", "| 2020–2022 | How should a generator consume and learn from retrieval? | RAG, FiD, FiD-KD, EMDR², RETRO, Atlas |\n", "| 2022–2024 | Can queries, timing, and evidence quality be controlled? | HyDE, FLARE, Self-RAG, CRAG, Adaptive-RAG |\n", "| 2025–2026 | Can retrieval become a learned reasoning action? | Search-R1, GRIP, Q-RAG, HiPRAG |" ] }, { "cell_type": "markdown", "metadata": { "tags": [ "binding-instrument", "publication-instrument" ] }, "source": [ "### Instrument check — what is actually bound here?\n", "\n", "These two observations inspect the current repository and dated source\n", "ledger. Run them after changing the evidence registry; if the counts or\n", "topic surface move, reread the conclusions that depended on them.\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "tags": [ "binding-instrument", "publication-instrument" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Repository root: resolved from the notebook location\n" ] } ], "source": [ "from pathlib import Path\n", "import sys\n", "\n", "ROOT = Path.cwd()\n", "if not (ROOT / \"src\").exists():\n", " ROOT = ROOT.parent\n", "sys.path.insert(0, str(ROOT / \"src\"))\n", "print(\"Repository root: resolved from the notebook location\")\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [ "binding-instrument", "publication-instrument" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Evidence cutoff: 2026-08-09\n", "Primary-source registry: 203 {'peer-reviewed': 181, 'preprint': 19, 'industry-report': 2, 'benchmark-program': 1}\n", "Most represented topic tags: [('benchmark', 26), ('multi-hop', 12), ('dense-retrieval', 12), ('generation', 9), ('efficiency', 9), ('reranking', 9), ('ann', 9), ('pretraining', 8), ('rag', 8), ('multimodal', 8), ('long-context', 8), ('graph', 8), ('evaluation', 8), ('memory', 7), ('reasoning', 7), ('reinforcement-learning', 7), ('citations', 7), ('embeddings', 7), ('retrieval', 6), ('distillation', 6)]\n" ] } ], "source": [ "import json\n", "from collections import Counter\n", "\n", "registry = json.loads((ROOT / \"research\" / \"sources.json\").read_text(encoding=\"utf-8\"))\n", "statuses = Counter(source[\"status\"] for source in registry[\"sources\"])\n", "topics = Counter(topic for source in registry[\"sources\"] for topic in source[\"topics\"])\n", "print(\"Evidence cutoff:\", registry[\"evidence_cutoff\"])\n", "print(\"Primary-source registry:\", len(registry[\"sources\"]), dict(statuses))\n", "print(\"Most represented topic tags:\", topics.most_common(20))\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "14 source documents → 14 traceable chunks\n", "8 labeled questions; evidence dates 2020-04-10 to 2026-07-01\n" ] } ], "source": [ "from pathlib import Path\n", "import sys\n", "\n", "ROOT = Path.cwd()\n", "if ROOT.name == 'notebooks':\n", " ROOT = ROOT.parent\n", "sys.path.insert(0, str(ROOT / 'src'))\n", "\n", "from rag_evolution.demo_data import demo_documents, demo_questions\n", "from rag_evolution.evaluation import aggregate_metrics, evaluate_retriever\n", "from rag_evolution.retrievers import BM25Retriever, HashingSemanticRetriever, HybridRetriever\n", "from rag_evolution.text import chunk_documents\n", "\n", "documents = demo_documents()\n", "questions = demo_questions()\n", "chunks = chunk_documents(documents, chunk_size=90, overlap=18)\n", "print(f'{len(documents)} source documents → {len(chunks)} traceable chunks')\n", "print(f'{len(questions)} labeled questions; evidence dates {documents[0].date} to {documents[-1].date}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stage 1 — BM25: a sparse baseline that never became obsolete\n", "\n", "For document length $|d|$, average length $\\operatorname{avgdl}$, term frequency $f(t,d)$, and parameters $k_1,b$:\n", "\n", "$$\\operatorname{BM25}(q,d)=\\sum_{t\\in q}\\operatorname{IDF}(t)\\frac{(k_1+1)f(t,d)}{f(t,d)+k_1(1-b+b|d|/\\operatorname{avgdl})}.$$\n", "\n", "Sparse retrieval is strong for names, codes, dates, and exact terminology. Its failure surface is vocabulary mismatch." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "QUERY: Which system used hard BM25 negatives?\n", " 1. dpr-2020 score=9.111\n", " 2. search-r1-2025 score=1.900\n", " 3. crag-2024 score=1.809\n", "\n", "QUERY: What did RAG-Token marginalize?\n", " 1. rag-2020 score=6.571\n", "\n", "QUERY: lookup external documents by meaning\n", " 1. atlas-2022 score=2.390\n", " 2. graphrag-2024 score=2.263\n", " 3. rag-2020 score=1.704\n" ] } ], "source": [ "bm25 = BM25Retriever(chunks)\n", "for query in [\n", " 'Which system used hard BM25 negatives?',\n", " 'What did RAG-Token marginalize?',\n", " 'lookup external documents by meaning',\n", "]:\n", " print(f'\\nQUERY: {query}')\n", " for result in bm25.search(query, 3):\n", " print(f' {result.rank}. {result.chunk.document_id:18s} score={result.score:.3f}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stage 2 — independently encoded semantic retrieval\n", "\n", "DPR made the dual-encoder recipe standard: $s(q,p)=E_Q(q)^\\top E_P(p)$, trained by contrastive loss with in-batch and hard negatives. Query and passage vectors can be indexed independently, unlike a cross-encoder. The proxy below preserves that contract and bridges a small declared synonym vocabulary, so it can run without downloading a model." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Semantic-proxy results for vocabulary-mismatch query:\n", " 1. dpr-2020 cosine=0.177\n", " 2. atlas-2022 cosine=0.171\n", " 3. grip-2026 cosine=0.137\n", " 4. rag-2020 cosine=0.136\n", " 5. fid-2021 cosine=0.101\n" ] } ], "source": [ "semantic = HashingSemanticRetriever(chunks, dimensions=512)\n", "query = 'lookup external documents by meaning'\n", "print('Semantic-proxy results for vocabulary-mismatch query:')\n", "for result in semantic.search(query, 5):\n", " print(f' {result.rank}. {result.chunk.document_id:18s} cosine={result.score:.3f}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stage 3 — hybrid candidate generation and reciprocal-rank fusion\n", "\n", "Sparse and semantic scores are not naturally calibrated. Reciprocal-rank fusion combines order rather than raw scale:\n", "\n", "$$\\operatorname{RRF}(d)=\\sum_r \\frac{w_r}{K+\\operatorname{rank}_r(d)}.$$\n", "\n", "This pattern reflects the empirical lesson from DPR, BEIR, SPLADE, and modern production retrieval: sparse and dense systems have complementary errors." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Document-level retrieval on the teaching questions:\n", "BM25 recall@5=1.000 MRR=1.000 nDCG@5=0.985\n", "semantic proxy recall@5=0.938 MRR=1.000 nDCG@5=0.952\n", "hybrid RRF recall@5=0.938 MRR=1.000 nDCG@5=0.952\n" ] } ], "source": [ "hybrid = HybridRetriever((('sparse', bm25, 1.0), ('semantic', semantic, 1.0)), rrf_constant=30)\n", "systems = {'BM25': bm25, 'semantic proxy': semantic, 'hybrid RRF': hybrid}\n", "\n", "def show_metrics(name, rows):\n", " mean = aggregate_metrics(rows)\n", " print(f\"{name:16s} recall@5={mean['recall@5']:.3f} MRR={mean['mrr']:.3f} nDCG@5={mean['ndcg@5']:.3f}\")\n", "\n", "print('Document-level retrieval on the teaching questions:')\n", "for name, system in systems.items():\n", " show_metrics(name, evaluate_retriever(system, questions, k=5))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stage 4 — retrieval-conditioned generation\n", "\n", "The original RAG paper optimized a truncated latent-document likelihood. RAG-Sequence chose one document for the output; RAG-Token marginalized documents per token. FiD instead encoded many passages independently and let one decoder fuse their representations. RETRO injected retrieved chunks during autoregressive pretraining; Atlas combined Contriever, FiD, pretraining, and reader-to-retriever distillation.\n", "\n", "These are not interchangeable: they differ in retrieval supervision, when retrieval occurs, passage count, integration point, index refresh, generator scale, and evaluation corpus." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-04-10 Dense Passage Retrieval (DPR) https://aclanthology.org/2020.emnlp-main.550/\n", "2020-05-22 Retrieval-Augmented Generation (RAG) https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html\n", "2020-07-02 Fusion-in-Decoder (FiD) https://aclanthology.org/2021.eacl-main.74/\n", "2021-12-08 RETRO https://proceedings.mlr.press/v162/borgeaud22a.html\n", "2022-08-05 Atlas https://jmlr.org/papers/v24/23-0037.html\n", "2023-10-17 Self-RAG https://openreview.net/forum?id=hSyW5go0v8\n", "2026-07-01 GRIP: Retrieval as Generation https://aclanthology.org/2026.acl-long.196/\n" ] } ], "source": [ "timeline_ids = ['dpr-2020', 'rag-2020', 'fid-2021', 'retro-2022', 'atlas-2022', 'self-rag-2023', 'grip-2026']\n", "by_id = {document.id: document for document in documents}\n", "for document_id in timeline_ids:\n", " document = by_id[document_id]\n", " print(f'{document.date} {document.title:38s} {document.source}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Interpretation\n", "\n", "1. Retrieval metrics diagnose candidate quality; they do not establish grounded answers.\n", "2. Dense retrieval is not a replacement for sparse retrieval. Hybrid retrieval is a strong default.\n", "3. More context is not monotonically better: FiD benefits from many passages, but Lost in the Middle and later evidence-utility work show distraction.\n", "4. Retrieval-conditioned generation does not guarantee causal attribution. The next notebook adds reranking, evidence budgeting, citations, abstention, and adaptive routing." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.9" }, "rag_evolution": { "presentation": "plain-field-notebook", "visual_version": 2 } }, "nbformat": 4, "nbformat_minor": 5 }