# Matryoshka embeddings and dimensions Some embedding models are trained so that the first N components of a vector are themselves a good embedding. This is called Matryoshka Representation Learning (MRL). With such a model you can keep a short prefix of each vector, and trade a small amount of quality for much less storage and faster search. Today you truncate on the client, as described in [Truncate on the client](#truncate-on-the-client), and map the field with the shorter dimension. The search engine also has a `truncate_dims` field option that does this server-side, described in [Server-side truncation](#server-side-truncation). > Use truncation only with MRL-trained models. For a model that was not trained this way, cutting > a vector short throws away information the model did not pack into the first components, and quality > drops sharply. ## Server-side truncation > **Coming soon:** `truncate_dims` is implemented in the search engine, but the public API does not > expose it yet. Collections are created from Elasticsearch-style mappings, which have no way to > express it. Until it is exposed, truncate on the client. This section describes the behavior so you > can plan for it, and the details may change before release. `truncate_dims` is an option on a vector field in the native schema: ```json { "fields": { "summary_v": { "type": "vector", "dims": 1024, "truncate_dims": 256, "metric": "cosine" } } } ``` | Property | Meaning | |---|---| | `dims` | The full dimension of the vectors your model produces. Required when you use `truncate_dims`. | | `truncate_dims` | The dimension that is stored and searched. Must be positive and smaller than `dims`. `0` clears a value inherited from a parent definition. | | `metric` | Must be `cosine`. Other metrics are rejected, see [Cosine only](#cosine-only). | The stored column, the approximate indexes, exact scans and full-precision rescoring all work on the shorter vector. ## What clients send Documents and queries can carry either of two lengths: - the full model output, `dims` components, which WarpSearch cuts to the first `truncate_dims`, or - a vector your provider already truncated to exactly `truncate_dims` components. You can mix the two, even within one multi-valued field. That means vectors you read back from the field (already truncated) can be written again without conversion, which is handy when you reindex. Any other length is rejected with a message that names both accepted sizes: ```text expected dims=1024 or truncate_dims=256, got 768 ``` This is deliberate. Once vectors are cut, vectors from two different models can have the same length, and the declared full size is what keeps another model's output out of the field. The check for finite numbers covers the whole input, including the part that is cut off. One limit follows from this: it cannot catch a model whose native size equals your `truncate_dims`. If you declare `dims: 3072, truncate_dims: 1536`, vectors from a 1536-dimension model are accepted as if they were already truncated, so keep one model per field. ## Cosine only `truncate_dims` requires `metric: cosine`. The reason is that the prefix of a unit vector is no longer unit length. For cosine, WarpSearch renormalizes the prefix on write, and normalizes the query at search time, so scores are correct. Under `l2` or `ip`, each document's shrunken prefix length would leak into the score, and MRL models are meant to be compared after renormalization anyway. Related rules: - With the default `normalize_on_write: true`, the stored prefix is renormalized. With `normalize_on_write: false` the raw prefix is stored and candidates are normalized at query time. - Declaring `normalized: true` together with `truncate_dims` is a schema error, because that promise cannot hold for a cut vector. - A vector whose prefix is all zeros has no cosine direction, even if the full vector does, and is skipped on write. - If you set `ann.pq.m`, it must divide `truncate_dims`, not `dims`. ## Models that support it | Model | How to get a shorter vector | Notes | |---|---|---| | OpenAI `text-embedding-3-small` and `-large` | The `dimensions` request parameter, or cut the output yourself | OpenAI reports a 256-dimension `text-embedding-3-large` embedding still beats the full 1536-dimension `text-embedding-ada-002` on MTEB. If you cut manually, renormalize. | | Google `gemini-embedding-001` | `output_dimensionality` | Recommended sizes are 768, 1536 and 3072. Outputs other than 3072 need manual normalization. | | `nomic-embed-text-v1.5` | Cut the output | The model card reports MTEB 62.28 at 768 dimensions, 61.04 at 256 and 59.34 at 128. Its reference code applies a layer norm before cutting, which WarpSearch does not do. | | `jina-embeddings-v3` | Provider `dimensions` option, or cut the output | Supports sizes from 1024 down to 32. | | `mxbai-embed-large-v1` | Cut the output | Supports MRL truncation. | Check your provider's current documentation for exact parameter names and supported sizes. ## Choosing N Storage and exact-scan cost scale linearly with the stored dimension. A 1024-dimension float32 vector takes 4,096 bytes. At 256 dimensions it takes 1,024 bytes, and at 128 it takes 512, about 4 times and 8 times less. In a test on 100,000 synthetic 1024-dimension vectors, an exact top-10 scan took about 13 to 16 ms at 1024 dimensions, about 3.6 ms at 256 and about 2 ms at 128. The latencies come from a shared machine and are noisy, but the proportions hold. Quality is the part you must measure on your data. Published figures to set expectations: - Hugging Face reports a Matryoshka model keeping about 98% of its performance at roughly 8% of the embedding size on the STS benchmark. - Vespa quotes about a 2% quality loss going from 1024 to 128 dimensions for MRL-trained models. - Combining a 512-dimension prefix with binary quantization has been reported to keep about 90% of retrieval accuracy, and 95 to 96% when candidates are rescored with floats. A reasonable process: 1. Embed a sample of your documents and some real queries with the full model. 2. Pick candidate sizes, for example 1024, 512, 256 and 128. 3. Compare recall and relevance against the full-length results, with exact search on. 4. Choose the smallest size whose quality you can accept. To compare several sizes from one indexing pass, declare truncated variants beside the full vector before you index. Each variant indexes the same input, and a query selects one by name, such as `summary_v__d256`. A variant added later covers only documents indexed after it. ```json "summary_v": { "type": "vector", "dims": 1024, "metric": "cosine", "variants": { "d256": { "type": "vector", "dims": 1024, "truncate_dims": 256, "metric": "cosine" } } } ``` ## Changing N later Changing the stored dimension means existing documents must be reindexed from your source vectors, the same as changing `dims`. WarpSearch cannot recover the cut-off components, so keep the full vectors (or the text to re-embed) outside the index. ## Truncate on the client This works with every API today. Cut the vector yourself. Take the first N components, then divide by the Euclidean norm so the vector has unit length. Do the same for queries, and map the field with `dims: N`. ```python import math def truncate(vec, n): head = vec[:n] norm = math.sqrt(sum(x * x for x in head)) or 1.0 return [x / norm for x in head] ``` Use the same N for every writer and every query. A client that truncates documents but forgets queries gets a dimension error at best. If two clients use different N, scores are silently wrong. The compatibility layer accepts vectors up to 8,192 dimensions per field. ## Combine with quantization Truncation and quantization multiply. Truncating 1024 to 256 dimensions cuts the float32 column four times. The approximate index is then built from the shorter vector, so its PQ or RaBitQ codes shrink too. Rescoring uses the stored prefix, not the full vector. See [Vector search](vector-search.md) for codecs, and for tuning recall once you have chosen a size.