01 · Foundation
What is an Embedding?
An embedding model maps text (or images, or code) to a fixed-length vector of floats— typically 256 to 3072 dimensions — such that inputs with similar meaning land near each other. That single property turns fuzzy questions like "which docs are about this?" into geometry: semantic search is nearest-neighbor search in embedding space.
The vector is not interpretable — dimension 512 does not mean "dog-ness". Meaning lives in the distances between vectors, not in any coordinate. Embed your corpus once, embed each query at request time, and rank by proximity.
Why This Beats Keywords
Keyword search
Misses synonyms and paraphrase
"k8s" never matches "Kubernetes"
No notion of topic or intent
Embedding search
Matches meaning, not spelling
Cross-lingual for free on many models
Same trick works for images and code
Where you meet them
RAG retrieval, semantic search, deduplication, clustering, recommendations, anomaly detection, classification-by-neighbor — all the same primitive: embed, then measure distance.
01 · Foundation
Similarity Math
Three distances cover practice. Cosine similarity compares direction only — the angle θ between vectors. Dot product is cosine scaled by magnitude, so longer vectors score higher. Euclidean is straight-line distance between the tips. On the unit-normalized vectors most APIs return, all three rank identically — the choice only matters for un-normalized embeddings.
Computing It
cosine.ts
// Cosine similarity: the angle between vectors, magnitude ignored
export function cosineSimilarity(a: number[], b: number[]): number {
let dot = 0;
let normA = 0;
let normB = 0;
for (let i = 0; i < a.length; i++) {
dot += a[i] * b[i];
normA += a[i] * a[i];
normB += b[i] * b[i];
}
return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}
// Most embedding APIs return unit-normalized vectors, so
// cosine == dot product, and euclidean ranks identically.
// Scores land in [-1, 1]; real text pairs cluster ~0.2-0.9,
// so compare scores to each other, never to an absolute bar.
Practical rules
Check whether your model normalizes (most do — then just use dot, it is the cheapest). Never mix vectors from different models or even different model versions in one index: their spaces are unrelated, and distances across them are noise.
01 · Foundation
Embedding Models 2026
The 2026 landscape splits by modality and deployment, not by one leaderboard. MTEB-style benchmarks are saturated and gamed — rankings on your own queries are the only ones that transfer. What did change materially: longer context per chunk, multimodal by default at the frontier, and matryoshka training making dimensions a runtime dial.
📝
Text
The workhorses: OpenAI text-embedding-3, Voyage, Cohere Embed, Gemini embedding. 1024–3072 dims, strong multilingual coverage.
🖼️
Multimodal
Text and images share one space — search screenshots, PDFs as pixels, and product photos with a text query.
💻
Code
Code-tuned models embed identifiers and structure, pairing natural-language queries with function bodies far better than general text models.
📏
Dimensions
More dims = more nuance, more RAM, slower search. 512–1024 is the practical sweet spot for most corpora.
🪆
Matryoshka
MRL-trained vectors front-load information: truncate 3072 → 512 dims and keep most quality. One model, many size/cost points.
🏠
Open weights
BGE, GTE, Nomic, Qwen-embedding run on your hardware — no per-token bill, no vendor lock on re-embedding.
The lock-in nobody prices in
Switching embedding models means
re-embedding the entire corpus — old and new vectors cannot be compared. Pin the exact model version, store the raw text alongside every vector, and budget for at least one full re-embed per year.
02 · Search
Nearest Neighbor Search
Finding the closest vectors is the whole game. Exact kNN compares the query to every vector — perfect recall, linear cost. Fine up to ~100k vectors; a brute-force scan of a million 1024-dim floats per query is where it stops being fine. ANN (approximate nearest neighbor) indexes trade a few percent of recall for orders-of-magnitude speedups, and HNSW is the index you will actually meet.
The Knobs That Matter
M · edges per nodeefConstruction · build effortefSearch · recall vs latencyrecall@k vs exact kNN
efSearch is the runtime dial: raise it until measured recall against an exact-kNN sample clears your bar (95%+ is typical), then stop paying latency for recall no one notices.
Filters break naive ANN
Post-filtering an ANN result ("top-10, then keep only tenant=42") can return zero rows even when matches exist. Use an index that supports
filtered search natively — the filter must participate in the graph traversal, not run after it.
02 · Search
Chunking Strategies
You rarely embed whole documents — you embed chunks, and chunking quality caps retrieval quality. Too big and the vector averages away the specific fact; too small and the chunk loses the context that made it meaningful. Most retrieval failures blamed on the model are chunking failures.
chunking.ts
type Chunk = { text: string; meta: { source: string; idx: number } };
// Fixed-size with overlap: the boring baseline that mostly works
export function chunkFixed(
text: string,
source: string,
size = 512, // target tokens; measure with your tokenizer
overlap = 64, // so a sentence cut at a boundary survives once
): Chunk[] {
const approxChars = size * 4; // ~4 chars per token
const step = (size - overlap) * 4;
const chunks: Chunk[] = [];
for (let start = 0, idx = 0; start < text.length; start += step, idx++) {
chunks.push({
text: text.slice(start, start + approxChars),
meta: { source, idx },
});
}
return chunks;
}
// Semantic chunking: split on headings / paragraphs first, then
// merge neighbors until the size budget is hit. Late chunking:
// embed the whole doc with a long-context model, then pool
// per-chunk token embeddings - each chunk keeps document context.
Picking a Strategy
📐
Fixed + overlap
Dumb, fast, predictable. 256–512 tokens with 10–15% overlap is the baseline to beat — often nothing beats it.
🧩
Semantic
Split on structure — headings, paragraphs, functions. Chunks align with how authors organized meaning.
🕰️
Late chunking
Embed the full document with a long-context model, pool token embeddings per chunk. Each chunk sees the whole doc.
Cheap wins
Prepend a breadcrumb to every chunk before embedding —
Handbook > Security > Access Control — so the vector carries its location. Store the source id and char offsets in metadata so you can cite, and fetch neighboring chunks at answer time when one chunk is not enough.
02 · Search
Hybrid Search & Reranking
Embeddings are great at paraphrase and terrible at exact tokens — part numbers, error codes, names. BM25 is the mirror image. Production systems run both and fuse: lexical + vector + a reranker beats either retriever alone on nearly every corpus, and it is the standard RAG retrieval stack in 2026.
🧭
Vector ANN
semantic top-100
Fusing Without Score Gymnastics
BM25 scores and cosine scores live on unrelated scales. Reciprocal Rank Fusion sidesteps calibration entirely by combining ranks instead of scores:
rrf.ts
// Reciprocal Rank Fusion: merge BM25 + vector result lists
// without calibrating their incompatible score scales
export function rrf(
lists: string[][], // ranked doc ids, one list per retriever
k = 60, // dampens the head of each list
): { id: string; score: number }[] {
const scores = new Map<string, number>();
for (const list of lists) {
list.forEach((id, rank) => {
scores.set(id, (scores.get(id) ?? 0) + 1 / (k + rank + 1));
});
}
return [...scores.entries()]
.map(([id, score]) => ({ id, score }))
.sort((a, b) => b.score - a.score);
}
// fuse top-100 from each retriever, keep the fused top ~50,
// then let a cross-encoder reranker pick the 5-10 that
// actually reach the model's context window.
Why the reranker earns its latency
Bi-encoders (embeddings) squeeze each text into one vector before ever seeing the query. A
cross-encoder reads query and document together and scores the pair directly — far more accurate, far too slow for the whole corpus. Retrieve wide and cheap, rerank narrow and precise.
03 · In Production
Vector Databases
The 2026 question is not "which vector DB is best" but whether you need a dedicated one at all. Under a few million vectors, the database you already run probably does this well enough — and keeps vectors transactionally consistent with the rows they describe.
🐘
pgvector
Postgres extension. Vectors next to your data, SQL filters, joins, transactions. The default answer below ~10M vectors.
🌲
Pinecone
Fully managed, serverless pricing, zero ops. You trade control and cost for never thinking about index tuning.
⚡
Qdrant
Open-source Rust engine. Strong native filtering, quantization built in, self-host or cloud.
🐡
Turbopuffer
Object-storage-native: vectors live on S3, cache on NVMe. Very cheap at rest — built for many mostly-idle namespaces.
🗄️
LanceDB
Embedded, file-based, columnar. SQLite-of-vectors ergonomics for local apps, notebooks, and edge deployments.
🔎
Search engines
Elasticsearch / OpenSearch / Vespa do BM25 and vectors in one place — one system for the whole hybrid stack.
What It Looks Like in Postgres
query.sql
-- pgvector: cosine distance operator is <=>
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE docs (
id bigserial PRIMARY KEY,
content text NOT NULL,
source text NOT NULL,
embedding vector(1024)
);
-- HNSW index; the opclass must match the query operator
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops);
-- Top-10 nearest neighbors with a metadata filter
SELECT id, content, 1 - (embedding <=> $1) AS similarity
FROM docs
WHERE source = 'handbook'
ORDER BY embedding <=> $1
LIMIT 10;
Selection heuristic
Already on Postgres and under ~10M vectors → pgvector. Hundreds of millions of vectors or strict p50 latency SLOs → dedicated engine. Zero-ops priority → managed. Local-first or embedded → LanceDB.
03 · In Production
Quantization & Cost
Vector search cost is mostly RAM: ANN indexes want the working set in memory, and float32 vectors are heavy. Quantization stores each dimension in fewer bits — int8 (4× smaller) or binary, one bit per dimension (32× smaller) — and recovers accuracy by rescoring the shortlist with full-precision vectors kept on disk.
memory-math.ts
// Memory for 10M vectors at 1024 dims
const vectors = 10_000_000;
const dims = 1024;
const float32 = vectors * dims * 4; // 40.96 GB
const int8 = vectors * dims * 1; // 10.24 GB (4x smaller)
const binary = (vectors * dims) / 8; // 1.28 GB (32x smaller)
// Typical recall, with full-precision rescoring of top candidates:
// int8 -> ~99% of float32 recall
// binary -> ~95-97% if you rescore the top-100 with full vectors
//
// Stack with matryoshka truncation (1024 -> 256 dims) for
// another 4x before quantization even starts.
The Standard Recipe
🪆
Truncate
matryoshka 1024 → 256
🗜️
Quantize
binary index in RAM
🎯
Rescore
float32 from disk → top-10
Do the arithmetic first
Embedding API calls are cheap and one-time; serving vectors is forever. A corpus that costs $50 to embed can cost hundreds a month to hold in float32 RAM. Quantization plus matryoshka routinely cuts serving cost 10–100× at a recall loss your eval suite — not your intuition — should sign off on.
03 · In Production
Failure Modes
Vector search fails quietly: it always returns the k nearest somethings, with confident-looking scores, whether or not anything relevant exists. Knowing the standard failure shapes is most of the defense.
Similar is not relevant
"How do I cancel my subscription?" embeds very close to "How do I renew my subscription?" — same words, same topic, opposite intent. Embeddings measure semantic proximity, not answerhood. This is precisely the gap a cross-encoder reranker closes; without one, near-miss retrievals become confidently wrong RAG answers.
Out-of-domain drift
A model trained on web prose scores 0.8 on everything when pointed at legal boilerplate, log lines, or medical shorthand — the whole corpus collapses into one blob and ranking degrades to noise. Benchmark on your domain before committing; fine-tuned or domain-specific models exist because this failure is common.
Stale index
The index only knows what you embedded. Edited docs whose vectors never updated are silent misinformation with a good similarity score. Treat embedding as a materialized view: update vectors in the same transaction or pipeline as the source write, and monitor index age against source-of-truth timestamps.
Detection Signals
top-1 score ≈ top-20 scorerecall@k dropping over timereranker overturns retriever ofteneval pairs from real user queries
A flat score curve means the query found no real neighborhood — better to say "no good match" than to hand the model five plausible-looking irrelevant chunks.
04 · Execution
Ship Checklist
The gap between a cosine-similarity demo and a retrieval system users trust is this list. Work through it before real queries arrive.
Retrieval readiness · 0/10
Model picked by benchmarking on YOUR queries, then pinned (name + version)
modelSame model + same preprocessing for documents and queries
modelChunking tuned with overlap; every chunk carries source metadata
ingestHybrid retrieval: BM25 + vector, fused with RRF
searchCross-encoder reranker on the fused top-50
searchMetadata filters applied inside the ANN query, not post-filtered
searchANN recall measured against exact kNN on a sample (target > 95%)
qualityQuantization evaluated with rescoring before paying for float32 RAM
costEval set of query → relevant-doc pairs runs in CI on every change
qualityRe-embed + reindex plan for model upgrades; stale-index monitoring on
opsWhere to start
If you adopt only three: build an eval set from real queries before tuning anything, add BM25 + RRF next to your vector search, and put a reranker in front of the context window. Those three fix the majority of "the AI retrieved the wrong thing" reports.