For AI agents: this page is also available as Markdown at /docs/matryoshka-and-dimensions.md. Every docs page is: add .md to its URL. llms.txt lists them all.

Matryoshka embeddings and dimensions

Some embedding models are trained so that the first N components of a vector are themselves a good embedding. This is called Matryoshka Representation Learning (MRL). With such a model you can keep a short prefix of each vector, and trade a small amount of quality for much less storage and faster search.

Today you truncate on the client, as described in Truncate on the client, and map the field with the shorter dimension. The search engine also has a truncate_dims field option that does this server-side, described in Server-side truncation.

Use truncation only with MRL-trained models. For a model that was not trained this way, cutting a vector short throws away information the model did not pack into the first components, and quality drops sharply.

#Server-side truncation

Coming soon: truncate_dims is implemented in the search engine, but the public API does not expose it yet. Collections are created from Elasticsearch-style mappings, which have no way to express it. Until it is exposed, truncate on the client. This section describes the behavior so you can plan for it, and the details may change before release.

truncate_dims is an option on a vector field in the native schema:

{
  "fields": {
    "summary_v": {
      "type": "vector",
      "dims": 1024,
      "truncate_dims": 256,
      "metric": "cosine"
    }
  }
}

The stored column, the approximate indexes, exact scans and full-precision rescoring all work on the shorter vector.

#What clients send

Documents and queries can carry either of two lengths:

• the full model output, dims components, which WarpSearch cuts to the first truncate_dims, or
• a vector your provider already truncated to exactly truncate_dims components.

You can mix the two, even within one multi-valued field. That means vectors you read back from the field (already truncated) can be written again without conversion, which is handy when you reindex.

Any other length is rejected with a message that names both accepted sizes:

expected dims=1024 or truncate_dims=256, got 768

This is deliberate. Once vectors are cut, vectors from two different models can have the same length, and the declared full size is what keeps another model's output out of the field. The check for finite numbers covers the whole input, including the part that is cut off.

One limit follows from this: it cannot catch a model whose native size equals your truncate_dims. If you declare dims: 3072, truncate_dims: 1536, vectors from a 1536-dimension model are accepted as if they were already truncated, so keep one model per field.

#Cosine only

truncate_dims requires metric: cosine. The reason is that the prefix of a unit vector is no longer unit length. For cosine, WarpSearch renormalizes the prefix on write, and normalizes the query at search time, so scores are correct. Under l2 or ip, each document's shrunken prefix length would leak into the score, and MRL models are meant to be compared after renormalization anyway.

Related rules:

• With the default normalize_on_write: true, the stored prefix is renormalized. With normalize_on_write: false the raw prefix is stored and candidates are normalized at query time.
• Declaring normalized: true together with truncate_dims is a schema error, because that promise cannot hold for a cut vector.
• A vector whose prefix is all zeros has no cosine direction, even if the full vector does, and is skipped on write.
• If you set ann.pq.m, it must divide truncate_dims, not dims.

#Models that support it

Check your provider's current documentation for exact parameter names and supported sizes.

#Choosing N

Storage and exact-scan cost scale linearly with the stored dimension. A 1024-dimension float32 vector takes 4,096 bytes. At 256 dimensions it takes 1,024 bytes, and at 128 it takes 512, about 4 times and 8 times less. In a test on 100,000 synthetic 1024-dimension vectors, an exact top-10 scan took about 13 to 16 ms at 1024 dimensions, about 3.6 ms at 256 and about 2 ms at 128. The latencies come from a shared machine and are noisy, but the proportions hold.

Quality is the part you must measure on your data. Published figures to set expectations:

• Hugging Face reports a Matryoshka model keeping about 98% of its performance at roughly 8% of the embedding size on the STS benchmark.
• Vespa quotes about a 2% quality loss going from 1024 to 128 dimensions for MRL-trained models.
• Combining a 512-dimension prefix with binary quantization has been reported to keep about 90% of retrieval accuracy, and 95 to 96% when candidates are rescored with floats.

A reasonable process:

1. Embed a sample of your documents and some real queries with the full model.
2. Pick candidate sizes, for example 1024, 512, 256 and 128.
3. Compare recall and relevance against the full-length results, with exact search on.
4. Choose the smallest size whose quality you can accept.

To compare several sizes from one indexing pass, declare truncated variants beside the full vector before you index. Each variant indexes the same input, and a query selects one by name, such as summary_v__d256. A variant added later covers only documents indexed after it.

"summary_v": {
  "type": "vector", "dims": 1024, "metric": "cosine",
  "variants": {
    "d256": { "type": "vector", "dims": 1024, "truncate_dims": 256, "metric": "cosine" }
  }
}

#Changing N later

Changing the stored dimension means existing documents must be reindexed from your source vectors, the same as changing dims. WarpSearch cannot recover the cut-off components, so keep the full vectors (or the text to re-embed) outside the index.

#Truncate on the client

This works with every API today. Cut the vector yourself. Take the first N components, then divide by the Euclidean norm so the vector has unit length. Do the same for queries, and map the field with dims: N.

import math

def truncate(vec, n):
    head = vec[:n]
    norm = math.sqrt(sum(x * x for x in head)) or 1.0
    return [x / norm for x in head]

Use the same N for every writer and every query. A client that truncates documents but forgets queries gets a dimension error at best. If two clients use different N, scores are silently wrong.

The compatibility layer accepts vectors up to 8,192 dimensions per field.

#Combine with quantization

Truncation and quantization multiply. Truncating 1024 to 256 dimensions cuts the float32 column four times. The approximate index is then built from the shorter vector, so its PQ or RaBitQ codes shrink too. Rescoring uses the stored prefix, not the full vector. See Vector search for codecs, and for tuning recall once you have chosen a size.

View this page as Markdown