An arXiv benchmark of 168 configurations per task separates the costs of hierarchical embeddings and quantization on MTEB tasks.
An arXiv benchmark of 168 configurations per task separates the costs of hierarchical embeddings and quantization on MTEB tasks.
A paper on arXiv reports an empirical benchmark of two efficiency techniques for text embeddings: hierarchical representations, which allow variable-length vectors, and quantization, which lowers numerical precision. The authors evaluated 38 tasks from the Massive Text Embedding Benchmark using four transformer-based embedding models. They combined 1D and 2D hierarchical representations with 32-bit, 8-bit, and binary quantization, testing 168 configurations per task. Storage was measured as theoretical per-vector size, alongside end-to-end evaluation time.
Under a Friedman test with Nemenyi post-hoc analysis, int8 quantization was statistically indistinguishable from float32 for pair classification and re-ranking and cost at most 1.5 percentage points overall. Binary quantization was significantly worse across every task category. Halving the embedding dimension once was not statistically detectable above m=16, but two successive halvings always were. Combining an 8x dimensionality reduction with int8 gave a 32x reduction in per-vector storage while retaining 94.4 percent of baseline effectiveness.
The excerpt does not name the four models, the authors, or the full set of task categories, and it does not report how the two techniques interact beyond the stated combinations, so claims about interaction effects cannot be verified here. The excerpt is also cut off mid-sentence about evaluation, leaving its timing findings incomplete. Readers should check the full paper for per-category breakdowns and hardware details before generalizing the results.
Source: Balancing hierarchical representation and quantization: A performance trade-off…