How Embedding Models Work

The Embedding Pipeline

Every embedding model follows the same basic pipeline: tokenize → encode → pool → normalize. Understanding each step helps you make better decisions about which model to use and how to prepare your data.

Step 1: Tokenization

Text is split into tokens — subword units that the model understands. "Embeddings" might become ["em", "bed", "ding", "s"]. Different models use different tokenizers:

Tokenization matters because it determines the model's context window — how many tokens it can process at once. Longer text = more tokens = potential truncation.

Step 2: Transformer Encoding

Each token is converted into a contextualized vector. The word "bank" gets a different vector in "river bank" versus "savings bank" because the transformer considers surrounding tokens through self-attention.

This is the critical difference from older approaches like Word2Vec, which gave "bank" the same vector regardless of context.

Step 3: Pooling

The transformer produces one vector per token, but we need a single vector for the entire input. Pooling strategies:

Step 4: Normalization

The output vector is scaled to unit length (magnitude = 1.0). This ensures cosine similarity equals dot product, simplifying and speeding up search.

Dimensionality: How Many Numbers?

Embedding dimension is a key design choice:

More dimensions = more nuance in meaning representation, but also more storage (each float32 = 4 bytes) and slower search.

Matryoshka embeddings (supported by OpenAI and some open models) let you truncate vectors to fewer dimensions with minimal quality loss. A 3,072-dim vector can be cut to 256 dims and still perform well for many tasks.

Model Comparison: What Matters

Quality Benchmarks (MTEB)

The Massive Text Embedding Benchmark ranks models across retrieval, classification, clustering, and reranking. Key findings:

Cost Comparison

For a million-document corpus at ~500 tokens each: OpenAI small costs $10, large costs $65, and local is free but requires GPU infrastructure.

Chunking Strategies

Since embedding models have context windows, long documents must be chunked — split into smaller pieces. This is one of the most impactful decisions in any embedding pipeline.

Common Strategies

Chunk Size Trade-offs

Overlap Matters

Adjacent chunks should share some text (overlap) so that information at chunk boundaries isn't lost. Typical overlap: 10-20% of chunk size.

Instruction-Tuned Embeddings

Modern embedding models accept instructions that tell the model what kind of similarity to capture:

# Without instruction
embed("Apple released new products")
→ vector near fruits AND tech companies

# With instruction: "Retrieve financial news"
embed("Retrieve financial news: Apple released new products")
→ vector near tech companies, far from fruits

Models like E5, Voyage, and Cohere embed-v3 support different input types (query vs document) that optimize the embedding for asymmetric search.

Key Takeaways