From Words to Vectors
The working page
This notebook makes the architectural history executable. It starts with BM25, adds an independently encoded semantic representation, fuses heterogeneous rankings, and connects those components to DPR, RAG, FiD, RETRO, and modern hybrid systems.
The implementation is intentionally offline and inspectable. BM25Retriever implements real Okapi BM25. HashingSemanticRetriever is a fixed-width synonym and character-ngram proxy for a dual encoder; it demonstrates the interface but is not a pretrained neural retriever or a paper reproduction.
Historical map
| Period | Core question | Representative answer |
|---|---|---|
| 1972–1995 | How should exact terms be weighted? | TF-IDF, relevance weighting, BM25 |
| 2014–2017 | Can raw facts be external memory? | Memory Networks, DrQA |
| 2019–2020 | Can retrieval be learned from QA/LM objectives? | ORQA, REALM, DPR |
| 2020–2022 | How should a generator consume and learn from retrieval? | RAG, FiD, FiD-KD, EMDR², RETRO, Atlas |
| 2022–2024 | Can queries, timing, and evidence quality be controlled? | HyDE, FLARE, Self-RAG, CRAG, Adaptive-RAG |
| 2025–2026 | Can retrieval become a learned reasoning action? | Search-R1, GRIP, Q-RAG, HiPRAG |
Instrument check — what is actually bound here?
These two observations inspect the current repository and dated source ledger. Run them after changing the evidence registry; if the counts or topic surface move, reread the conclusions that depended on them.
from pathlib import Path
import sys
ROOT = Path.cwd()
if not (ROOT / "src").exists():
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / "src"))
print("Repository root: resolved from the notebook location")
Repository root: resolved from the notebook location
import json
from collections import Counter
registry = json.loads((ROOT / "research" / "sources.json").read_text(encoding="utf-8"))
statuses = Counter(source["status"] for source in registry["sources"])
topics = Counter(topic for source in registry["sources"] for topic in source["topics"])
print("Evidence cutoff:", registry["evidence_cutoff"])
print("Primary-source registry:", len(registry["sources"]), dict(statuses))
print("Most represented topic tags:", topics.most_common(20))
Evidence cutoff: 2026-08-09
Primary-source registry: 203 {'peer-reviewed': 181, 'preprint': 19, 'industry-report': 2, 'benchmark-program': 1}
Most represented topic tags: [('benchmark', 26), ('multi-hop', 12), ('dense-retrieval', 12), ('generation', 9), ('efficiency', 9), ('reranking', 9), ('ann', 9), ('pretraining', 8), ('rag', 8), ('multimodal', 8), ('long-context', 8), ('graph', 8), ('evaluation', 8), ('memory', 7), ('reasoning', 7), ('reinforcement-learning', 7), ('citations', 7), ('embeddings', 7), ('retrieval', 6), ('distillation', 6)]
from pathlib import Path
import sys
ROOT = Path.cwd()
if ROOT.name == 'notebooks':
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / 'src'))
from rag_evolution.demo_data import demo_documents, demo_questions
from rag_evolution.evaluation import aggregate_metrics, evaluate_retriever
from rag_evolution.retrievers import BM25Retriever, HashingSemanticRetriever, HybridRetriever
from rag_evolution.text import chunk_documents
documents = demo_documents()
questions = demo_questions()
chunks = chunk_documents(documents, chunk_size=90, overlap=18)
print(f'{len(documents)} source documents → {len(chunks)} traceable chunks')
print(f'{len(questions)} labeled questions; evidence dates {documents[0].date} to {documents[-1].date}')14 source documents → 14 traceable chunks 8 labeled questions; evidence dates 2020-04-10 to 2026-07-01
Stage 1 — BM25: a sparse baseline that never became obsolete
For document length \(|d|\), average length \(\operatorname{avgdl}\), term frequency \(f(t,d)\), and parameters \(k_1,b\):
Sparse retrieval is strong for names, codes, dates, and exact terminology. Its failure surface is vocabulary mismatch.
bm25 = BM25Retriever(chunks)
for query in [
'Which system used hard BM25 negatives?',
'What did RAG-Token marginalize?',
'lookup external documents by meaning',
]:
print(f'\nQUERY: {query}')
for result in bm25.search(query, 3):
print(f' {result.rank}. {result.chunk.document_id:18s} score={result.score:.3f}')QUERY: Which system used hard BM25 negatives? 1. dpr-2020 score=9.111 2. search-r1-2025 score=1.900 3. crag-2024 score=1.809 QUERY: What did RAG-Token marginalize? 1. rag-2020 score=6.571 QUERY: lookup external documents by meaning 1. atlas-2022 score=2.390 2. graphrag-2024 score=2.263 3. rag-2020 score=1.704
Stage 2 — independently encoded semantic retrieval
DPR made the dual-encoder recipe standard: \(s(q,p)=E_Q(q)^\top E_P(p)\), trained by contrastive loss with in-batch and hard negatives. Query and passage vectors can be indexed independently, unlike a cross-encoder. The proxy below preserves that contract and bridges a small declared synonym vocabulary, so it can run without downloading a model.
semantic = HashingSemanticRetriever(chunks, dimensions=512)
query = 'lookup external documents by meaning'
print('Semantic-proxy results for vocabulary-mismatch query:')
for result in semantic.search(query, 5):
print(f' {result.rank}. {result.chunk.document_id:18s} cosine={result.score:.3f}')Semantic-proxy results for vocabulary-mismatch query: 1. dpr-2020 cosine=0.177 2. atlas-2022 cosine=0.171 3. grip-2026 cosine=0.137 4. rag-2020 cosine=0.136 5. fid-2021 cosine=0.101
Stage 3 — hybrid candidate generation and reciprocal-rank fusion
Sparse and semantic scores are not naturally calibrated. Reciprocal-rank fusion combines order rather than raw scale:
This pattern reflects the empirical lesson from DPR, BEIR, SPLADE, and modern production retrieval: sparse and dense systems have complementary errors.
hybrid = HybridRetriever((('sparse', bm25, 1.0), ('semantic', semantic, 1.0)), rrf_constant=30)
systems = {'BM25': bm25, 'semantic proxy': semantic, 'hybrid RRF': hybrid}
def show_metrics(name, rows):
mean = aggregate_metrics(rows)
print(f"{name:16s} recall@5={mean['recall@5']:.3f} MRR={mean['mrr']:.3f} nDCG@5={mean['ndcg@5']:.3f}")
print('Document-level retrieval on the teaching questions:')
for name, system in systems.items():
show_metrics(name, evaluate_retriever(system, questions, k=5))Document-level retrieval on the teaching questions: BM25 recall@5=1.000 MRR=1.000 nDCG@5=0.985 semantic proxy recall@5=0.938 MRR=1.000 nDCG@5=0.952 hybrid RRF recall@5=0.938 MRR=1.000 nDCG@5=0.952
Stage 4 — retrieval-conditioned generation
The original RAG paper optimized a truncated latent-document likelihood. RAG-Sequence chose one document for the output; RAG-Token marginalized documents per token. FiD instead encoded many passages independently and let one decoder fuse their representations. RETRO injected retrieved chunks during autoregressive pretraining; Atlas combined Contriever, FiD, pretraining, and reader-to-retriever distillation.
These are not interchangeable: they differ in retrieval supervision, when retrieval occurs, passage count, integration point, index refresh, generator scale, and evaluation corpus.
timeline_ids = ['dpr-2020', 'rag-2020', 'fid-2021', 'retro-2022', 'atlas-2022', 'self-rag-2023', 'grip-2026']
by_id = {document.id: document for document in documents}
for document_id in timeline_ids:
document = by_id[document_id]
print(f'{document.date} {document.title:38s} {document.source}')2020-04-10 Dense Passage Retrieval (DPR) https://aclanthology.org/2020.emnlp-main.550/ 2020-05-22 Retrieval-Augmented Generation (RAG) https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html 2020-07-02 Fusion-in-Decoder (FiD) https://aclanthology.org/2021.eacl-main.74/ 2021-12-08 RETRO https://proceedings.mlr.press/v162/borgeaud22a.html 2022-08-05 Atlas https://jmlr.org/papers/v24/23-0037.html 2023-10-17 Self-RAG https://openreview.net/forum?id=hSyW5go0v8 2026-07-01 GRIP: Retrieval as Generation https://aclanthology.org/2026.acl-long.196/
Interpretation
- Retrieval metrics diagnose candidate quality; they do not establish grounded answers.
- Dense retrieval is not a replacement for sparse retrieval. Hybrid retrieval is a strong default.
- More context is not monotonically better: FiD benefits from many passages, but Lost in the Middle and later evidence-utility work show distraction.
- Retrieval-conditioned generation does not guarantee causal attribution. The next notebook adds reranking, evidence budgeting, citations, abstention, and adaptive routing.