Right-sized embeddings make it practical to use larger AI models for stronger search results while reducing the compute and memory footprint of the retrieval system. Although larger models can capture more nuance, their larger vectors increase storage, memory, bandwidth, and processing requirements. Matryoshka Representation Learning (MRL) addresses this trade-off by allowing a vector truncated to fewer dimensions to function as a complete embedding, preserving search quality while lowering infrastructure requirements, supporting more searches on existing infrastructure, and improving compatibility with vector database limits.
Useful retrieval information can be concentrated within a smaller part of the embedding space.
In a nutshell, the empirical evidence suggests that larger MRL-capable embedding models can deliver strong retrieval quality while using a smaller stored vector that fits common production constraints. Across our financial retrieval evaluations, every 1,024-dimensional output retained at least 97.3% of its model's 2,048-dimensional Recall@10 score. For example, the 1,024-dimensional output from the NVIDIA Nemotron-3-Embed-8B-BF16 model halved the raw vector payload, increased measured peak throughput by 13.7%, and fit comfortably within pgvector's standard full-precision index limit.
Understanding embedding dimensions
An embedding acts as a numerical fingerprint of meaning. By converting questions and documents into points on the same semantic map, an embedding model allows a retrieval system to find documents that are closest in meaning to a question.
This semantic matching supports search, recommendation, agent memory, and retrieval-augmented generation (RAG). If retrieval misses the right evidence, a downstream language model cannot reliably compensate for the missing information.
Search can match meaning even when a question and a document use different words.
Additional dimensions can preserve more detail, but they also increase the size of every stored vector. The practical question is therefore not how large an embedding a model can produce, but how many dimensions a particular retrieval workload actually needs.
How Matryoshka Representation Learning helps
MRL addresses this trade-off by training an embedding to remain useful when it is truncated to fewer dimensions. The structure resembles a set of nested containers, with the full vector available alongside several useful truncated versions.
An MRL-capable model can expose several deployment-ready vector sizes without retraining the model.
This structure gives teams practical flexibility:
- Use a larger vector when maximum measured retrieval quality is the priority.
- Use a balanced output when database efficiency and quality both matter.
- Use a compact output when the vector index has a strict dimensional limit.
Research suggests that this opportunity extends beyond MRL. A recent study of embedding truncation found that, in some scenarios, even non-MRL embeddings can remain competitive after moderate random truncation. MRL makes the approach easier to apply because the model is explicitly trained to produce useful truncated vectors.
The ideal cutoff still depends on the workload because financial search, product discovery, legal retrieval, and enterprise knowledge systems may tolerate different quality trade-offs. Instead of choosing between an oversized index and an entirely different model, teams can benchmark several output dimensions from one capable model.
What the financial retrieval evaluations showed
Financial Question Answering (FIQA) is a public financial retrieval dataset in the Benchmarking Information Retrieval (BEIR) collection. It is widely used because it brings together natural financial questions, a realistic passage collection, and human relevance judgments in a reproducible benchmark. Its test split contains 648 questions and roughly 57,600 passages.
We measured two common retrieval metrics:
- Recall@10 reports how many relevant passages appear in the first ten results.
- Normalized Discounted Cumulative Gain at 10 (nDCG@10) measures ranking quality by assigning higher scores when the most relevant passages appear nearer the top of the first ten results.
To isolate the effect of the embedding model and vector dimensions, we evaluated all 648 questions against 57,600 ingested passages using exact cosine retrieval without a reranker. This prevents a second-stage model from obscuring the differences being measured.
FIQA English retrieval evaluation using exact embedder-only search. Higher is better.
At both tested dimension settings, Nemotron led Qwen/Qwen3-Embedding-8B and Qwen/Qwen3-Embedding-4B. When its output was reduced from 2,048 to 1,024 dimensions, Recall@10 decreased by only 0.0191 and nDCG@10 decreased by 0.0155.
Across the three models, every 1,024-dimensional output retained at least 97.3% of its model's 2,048-dimensional Recall@10. Nemotron retained 97.4%, while Qwen3-Embedding-8B retained 98.3%. Qwen3-Embedding-4B improved slightly at 1,024 dimensions, with Recall@10 increasing by 0.0019 and nDCG@10 increasing by 0.0022.
To make the result more tangible, if the 2,048-dimensional Nemotron system found about 72 of every 100 relevant items in the top ten results, the 1,024-dimensional system found about 70. In other words, half the dimensions preserved nearly all of the measured quality because the useful ranking signal was sufficiently concentrated.
Half the vector payload
Pgvector stores a full-precision vector in roughly 4 × dimensions + 8 bytes. Reducing a vector from 2,048 to 1,024 dimensions therefore cuts the raw vector payload almost exactly in half.
At one million embeddings, the estimated raw payload falls from about 8.20 gigabytes (GB) to 4.10 GB. At ten million embeddings, the saving is roughly 41 GB for every stored copy before row metadata, alignment, and index graph links are counted.
Smaller vectors also reduce pressure on caches, memory bandwidth, backups, replication, and service responses.
These savings multiply across an operating environment. Production systems may keep replicas, backups, staging copies, and cached representations, so reducing the base vector size benefits every copy.
A clean fit for pgvector indexing
Hierarchical Navigable Small World (HNSW) indexes enable fast approximate nearest-neighbor search at scale. In pgvector, an HNSW index using the full-precision vector type currently supports up to 2,000 dimensions.
This limit creates an awkward boundary for common model outputs:
- A 2,048-dimensional vector exceeds the full-precision limit by 48 dimensions.
- A native 4,096-dimensional vector exceeds it by much more.
- A 1,024-dimensional MRL output stays safely inside the standard path.
The 1,024-dimensional output retains full precision and leaves substantial room below pgvector's 2,000-dimension HNSW limit.
Pgvector also provides the halfvec type, which supports HNSW indexes up to 4,000 dimensions. Although it offers another option, it uses a different precision path and still cannot accept a native 4,096-dimensional vector. Models that support dynamic slicing may also be able to use a vector truncated to 2,000 dimensions.
Fewer dimensions, less computation, and higher measured throughput
Cosine similarity and dot product calculations must process every dimension in an embedding vector, so their computational complexity scales linearly with the number of dimensions. This relationship is expressed as O(d), where d is the number of dimensions. A 1,024-dimensional vector therefore requires half the arithmetic work of a 2,048-dimensional vector and one quarter of the work of a 4,096-dimensional vector.
The FIQA retrieval campaign measured peak throughput of 31.5 queries per second at 1,024 dimensions, up from 27.7 at 2,048 dimensions.
The FIQA retrieval evaluations reflected this relationship. When we reduced the vector from 2,048 to 1,024 dimensions, measured peak throughput increased by 13.7%, from 27.7 to 31.5 queries per second. The exact gain will vary in production because HNSW latency also depends on graph traversal, cache behavior, filtering, concurrency, and model serving. Even so, reducing the dimensions removes computation from every similarity calculation and creates more capacity in the retrieval path.
What becomes smaller and what does not
MRL reduces the size of the retrieval representation, but it does not reduce the model required to create that representation.
| Gets smaller | Operational benefit | Does not shrink | Why |
|---|---|---|---|
| Stored vectors | Less database, cache, backup, and replication traffic | Model weights and accelerator memory | The full model still performs the encoding |
| Similarity computation | Fewer arithmetic operations | Text-encoding compute | Slicing happens after the model creates its representation |
| Service payloads | Reduced vector payload size per embedding | Retrieval quality risk | Truncating the vector can reduce retrieval quality, so each dimensionality must be validated for the target workload |
Running the 8-billion-parameter Nemotron model still requires an estimated 16 GB of accelerator memory. MRL reduces storage, transfer, and retrieval computation after an embedding has been generated, while leaving the model weights and encoding workload unchanged.
Conclusion
A larger embedding model does not necessarily require a larger vector index. When useful retrieval information is concentrated in fewer dimensions, MRL provides a practical way to expose a compact representation without giving up most of the measured quality.
Across the FIQA retrieval evaluations, every 1,024-dimensional output retained nearly all of its model's measured 2,048-dimensional recall. Nemotron combined the strongest retrieval quality in this study with a compact output that fits pgvector's standard HNSW path. Taken together, the results support a straightforward operating pattern: use a capable model, store a right-sized vector, and validate the cutoff for the target domain.