The Evidence Path

Retrieval pretraining and non-parametric memory

4 of 12 · Training · 7 named systems

Retrieval-augmented learning systems place external memory at pretraining, token prediction, chunk attention, or frozen-model inference.

Retrieval pretraining and non-parametric memory: architecture comparisonEach lane reads from input to output.

Latent retrieval pretraining

  1. maskTraining target
  2. retrieveLatent evidence
  3. predictConditioned loss
  4. refreshIndex update
  • MARGE
  • REALM
  • Atlas

Token memory

  1. stateHidden state
  2. lookupNearest contexts
  3. interpolateToken distribution
  4. decodeNext token
  • kNN-LM
  • SPALM

Chunk memory

  1. chunkInput window
  2. retrieveNeighbor chunks
  3. attendCross-attention
  4. predictContinuation
  • RETRO

Frozen generator

  1. retrieveCandidate contexts
  2. scoreGenerator feedback
  3. trainRetriever only
  4. inferBlack-box generation
  • REPLUG

Read the full explanation