RAG Fundamentals

Retrieval-Augmented Generation splits into two pipelines. Everything else in RAG engineering is tuning one of these stages.

INGEST:  load → chunk → embed → store
QUERY:   embed query → retrieve top-k → rerank → generate with citations

The empirical rule: retrieval is the bottleneck. If the right chunk isn't in the context, no amount of prompt engineering saves the generation step. Debug retrieval first, always.

Chunking

Baseline: 256–512 token windows with 10–20% overlap so facts straddling boundaries survive.
Structure-aware beats naive: split on headings, paragraphs, function boundaries — a fixed-width splitter that bisects a table destroys its meaning.
Semantic chunking: embed adjacent sentences; split where similarity between neighbors drops (a topic shift detector).
Anthropic contextual retrieval: before embedding, prepend an LLM-generated situating sentence to each chunk ("This chunk is from the 2024 10-K, discussing Q3 revenue…"). Chunks become self-describing, which sharply cuts failed retrievals for context-dependent text. Pairs naturally with prompt-caching — cache the full document while generating per-chunk context cheaply.

Hybrid search

Dense vectors capture meaning; BM25 (lexical) captures exact tokens — IDs, error codes, function names that embeddings blur. Run both and fuse with Reciprocal Rank Fusion:

RRF score(d) = Σ over arms  1 / (k + rank_arm(d)),   k ≈ 60

Rank-based fusion sidesteps incomparable score scales between arms. Representative benchmark: hybrid ≈ 66.4% MRR vs ≈ 56.7% for semantic-only. Implementation on Postgres in supabase-pgvector.

Reranking

	Bi-encoder	Cross-encoder
Encodes	query and doc separately	query + doc together
Relevance signal	vector distance	full token-level attention
Speed	fast (precomputable)	slow (per-pair inference)
Role	first-stage retrieval	second-stage rerank

Recipe: retrieve ~20 candidates cheaply, cross-encoder rerank, keep top 3–5 for the context window. You buy cross-encoder accuracy while only paying for it on 20 pairs.

Evaluation

Measure the two stages separately — a bad answer can come from either:

Retrieval metrics: recall@k (did any relevant doc make top-k?), MRR (mean reciprocal rank of first relevant hit), nDCG (graded, position-discounted relevance).
Generation metrics: faithfulness/groundedness — is every claim in the answer supported by the retrieved context?

High faithfulness with low recall = confidently answering from the wrong documents.

Beyond single-shot

Query rewriting: an LM reformulates the raw user query (expand acronyms, split multi-part questions, strip chit-chat) before embedding.
Agentic RAG: retrieval as a tool in an agent loop — the model decides when to search, issues multiple queries, inspects results, and re-queries. STORM's TopicExpert (storm-pipeline) is exactly this shape. See also rag-patterns and agentic-patterns.

Gotchas

Same embedding model for query and corpus. Different models produce incompatible spaces; similarity scores become noise.
Dimension mismatch is a hard error: a vector(1536) column rejects a 384-dim embedding at insert/query time. Migrating embedding models means re-embedding the corpus.

Key terms

RAG (Retrieval-Augmented Generation) — architecture that retrieves relevant documents at query time and conditions generation on them, with citations.
Chunking — splitting documents into embeddable units; baseline 256–512 tokens with 10–20% overlap, structure-aware where possible.
Semantic chunking — splitting where adjacent-sentence embedding similarity drops, approximating topic boundaries.
Contextual retrieval — Anthropic's technique of prepending an LLM-generated situating sentence to each chunk before embedding.
BM25 — classic lexical ranking function; the keyword arm of hybrid search, strong on exact identifiers embeddings miss.
Reciprocal Rank Fusion (RRF) — rank-based fusion, score = Σ 1/(k + rank) with k ≈ 60, for combining dense and lexical result lists.
Cross-encoder — reranker that attends over query and document jointly; accurate but per-pair expensive, so used only on a shortlist.
Bi-encoder — embeds query and documents independently for fast vector-distance retrieval; the first-stage workhorse.
recall@k / MRR / nDCG — retrieval-quality metrics: any-hit rate in top k, reciprocal rank of first hit, and graded position-discounted gain.
Faithfulness (groundedness) — generation-quality metric: whether every generated claim is supported by the retrieved context.
Agentic RAG — retrieval exposed as a tool inside an agent loop; the model decides when and what to search, iteratively.
Query rewriting — LM reformulation of the user query before embedding to improve retrieval hit rate.