Skip to content

Chapter 2.8 — Embedding dimensionality, Matryoshka and retrieval cost in production

🎯 Objective

Treat embedding dimensionality as an architectural decision with effects on storage, RAM, latency, indexing cost, recall quality and multi-tenant isolation. Establish when it is safe to reduce dimension, which methods preserve ranking, and how to evaluate it.

🧠 Core concept

An embedding is a d-dimensional vector. d (the "dimension") is a hyperparameter of the embedding model (text-embedding-3-small, nomic-embed-text, bge-large, etc.). Typical values range from 256 to 4096+.

Each vector stores d × bytes_per_value. In production, this multiplies by:

  • the number of indexed chunks (N);
  • the number of copies (replicas, backups, parallel indexes);
  • the ANN structure (HNSW usually adds 30–60% overhead);
  • the number of tenants/collections, if each has its own index.

🧮 Basic arithmetic

For a corpus of N chunks with vectors of dimension d in FP32:

weights_storage = N × d × 4 bytes
ANN_RAM (HNSW) ≈ weights_storage × (1.3 to 1.6) + links_overhead

Example: 10 million chunks, d = 1536, FP32 -> 10⁷ × 1536 × 4 = ~61 GB of vectors alone; HNSW pushes it to ~85–95 GB. Reducing to d = 512 cuts it to ~20 GB; quantizing values to int8 drops it to ~5 GB.

🧠 Paths to reduce dimension

Method What it does Cost to apply Preserves ranking?
Simple truncation Keeps the first k components Almost zero Generally no. A classic embedding is not trained for this.
PCA / SVD Linear projection over a representative dataset Medium (needs sample + recompute) Reasonable for analysis; worsens ranking on a heterogeneous corpus
Random projection Multiplies by a random matrix Low Approximate (Johnson-Lindenstrauss); useful in pre-filtering
Model trained at low dimension Train another model directly at smaller d High (training/model selection) Good, but requires re-embedding and re-evaluation
Matryoshka Representation Learning (MRL) The model is trained so that prefixes of d (e.g., 64, 128, 256, 512, 1024) are also semantically valid Low at the application (just truncate) Yes, within the trained levels
Value quantization (FP32->FP16/INT8/binary) Reduces bits per component Low to medium Good in INT8; binary loses more

🧠 Matryoshka Representation Learning

MRL (Kusupati et al., 2022) trains embeddings with a loss that forces each prefix of the vector to already be a useful representation. Result: the same model serves embeddings of, for example, 64, 128, 256, 512, 1024 and 1536 dimensions — just truncate.

Practical implications:

  • Multi-resolution indexing. You can store full embeddings and serve fast results with small prefixes for a first phase, then refine with larger prefixes on the top-k.
  • Controlled cost. Smaller tenants or less critical tasks use smaller d; critical tasks use the full d.
  • No re-embedding. Changing d in production does not require reindexing everything if the model is Matryoshka — just truncate and revalidate.

Known implementations: OpenAI's text-embedding-3-small/large support the dimensions parameter (shortening); nomic-embed-text v1.5 and later, and Sentence Transformers models, expose Matryoshka variants.

⚠️ Maturity. MRL is a recent technique, widely adopted by providers (OpenAI, Nomic, Cohere) and by Sentence Transformers. It is not universal — old models are not Matryoshka and truncating them comes out worse. Always check the model's documentation.

🏗️ How this shows up in production

  • Storage and RAM. In RAG with tens of millions of chunks, dimension is directly a monthly cost. Reducing from 1536 to 768 can save a significant digit of infrastructure.
  • Retrieval latency. ANN with smaller vectors is faster (memory, CPU cache, SIMD). In HNSW, search time grows roughly with O(d × log N × ef).
  • Initial indexing cost. Reindexing 100M chunks is expensive; having Matryoshka avoids re-embedding when changing dimension.
  • Multi-tenant RAG. Small tenants can use a smaller dimension and shared indexes with a tenant_id filter; large tenants can have dedicated indexes at a high dimension. Per-tenant isolation remains a mandatory control, not a property of dimension.
  • Vector DB and ANN (HNSW/IVF). HNSW favors recall with higher M and efSearch; IVF favors scale with nprobe. In both, a larger dimension costs more, and the quality gain is not linear.
  • Hybrid search and reranker. When there is a strong reranker (cross-encoder), the embedding dimension can be smaller without major loss — the reranker recovers precision. When there is no reranker, a high dimension matters more for the top-k.
  • Metadata filtering. Filters (tenant, language, jurisdiction) are mandatory and independent of dimension. Bad filters destroy any embedding gain.

⚖️ Trade-offs

Axis Increase d Reduce d
Recall@k on a difficult corpus Better (up to saturation) Worse on non-Matryoshka models
Storage and RAM Worse linearly Better linearly
ANN latency Worse Better
Re-embedding cost High High if switching models
Robustness to bad chunk Little effect Little effect
Robustness to ambiguous query Small effect; reranker matters more Small effect

🚨 Failure modes

  • Truncating embeddings of a model not trained on Matryoshka and assuming similar recall.
  • Reducing dimension and measuring only MRR or NDCG on a small corpus; in production, the loss shows up on rare queries and on multi-tenant.
  • Swapping the embedding model while keeping the old index. Old and new vectors coexist and produce chaotic ranking.
  • Forgetting tenant filters: cross-tenant leakage is independent of dimension.
  • Evaluating dimension without a reranker when production has a reranker, or vice versa.

🛡️ Controls and mitigations

  • A retrieval eval set with golden queries per tenant, language and question type. Metrics: recall@k, MRR, NDCG, downstream groundedness.
  • Dimension × quality curve. Evaluate 256, 512, 768, 1024 and full dimension on the same eval set. Look for the "knee of the curve" dimension.
  • Versioned reindexing. An embedding has model_id + version + dimensions. The old index is kept until the new one passes the gates.
  • Value quantization combined with Matryoshka. E.g., int8 × d=512 can beat fp32 × d=1024 in quality-per-byte.
  • Mandatory tenant filters at the storage level, not in the prompt.

📈 Metrics

  • storage_per_million_chunks (GB).
  • ram_for_index (GB) per ANN configuration.
  • p95_retrieval_latency_ms per dimension.
  • recall@k, mrr, ndcg per tenant/language.
  • groundedness and citation_coverage of the final answer (downstream).
  • cost_per_thousand_embeddings (provider or self-host).
  • projected reembedding_cost if you switch models.

🧪 How to evaluate the ideal dimension

  1. Build a representative golden set (hundreds to thousands of queries).
  2. Embed the corpus at the largest supported dimension.
  3. Evaluate recall@k, MRR and NDCG by truncating to 256, 512, 768, 1024 and full dimension, with and without a reranker.
  4. Evaluate groundedness on the final answer.
  5. Choose the smallest dimension whose quality does not drop beyond an accepted delta (e.g., ≤ 1 absolute NDCG point).
  6. Document the decision in an ADR and pin dimensions in the index contract.

📌 Checklist for choosing an embedding dimension in production

  • [ ] Is the model Matryoshka? If not, consider that reducing is not free.
  • [ ] Is there an eval set per tenant/language with at least hundreds of queries?
  • [ ] Is there a documented dimension × quality curve?
  • [ ] Were storage and RAM estimated for the expected corpus growth?
  • [ ] Is a reranker in the pipeline? Did the dimension account for that effect?
  • [ ] Do model_id + version + dimensions + index_version appear in tracing?
  • [ ] Are mandatory tenant filters in storage, not in the prompt?
  • [ ] Is there a re-embedding procedure with a parallel index + dual-read?
  • EX-LLM-09 — comparison of dimensions (256/512/1024) with Matryoshka and a reranker; quality × cost curve.
  • EX-LLM-10 — safe re-embedding with a parallel index (dual-write + dual-read).

📚 References