MTEB: Massive Text Embedding Benchmark

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037 May 2-6, 2023 ©2023 Association for Computational Linguistics MTEB: Massive Text Embedding Benchmark Niklas Muennighoff1, Nouamane Tazi1, Loïc Magne1, Nils Reimers2* 1Hugging Face 2cohere.ai 1firstname@huggingface.co 2info@nils-reimers.de Abstract Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art em- beddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper eval- uation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a to- tal of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a pub- lic leaderboard at https://github.com/ embeddings-benchmark/mteb. 1 Introduction Natural language embeddings power a variety of use cases from clustering and topic representa- tion (Aggarwal and Zhai, 2012; Angelov, 2020) to search systems and text mining (Huang et al., 2020; Zhu et al., 2021; Nayak, 2019) to feature representations for downstream models (Saharia et al., 2022; Borgeaud et al., 2022). Using gener- ative language models or cross-encoders for these applications is often intractable, as they may re- quire exponentially more computations (Reimers and Gurevych, 2019). However, the evaluation regime of current text embedding models rarely covers the breadth of their possible use cases. For example, Sim- CSE (Gao et al., 2021b) or SBERT (Reimers and *Most of the work done while at Hugging Face Gurevych, 2019) solely evaluate on STS and clas- sification tasks, leaving open questions about the transferability of the embedding models to search or clustering tasks. STS is known to poorly corre- late with other real-world use cases (Neelakantan et al., 2022; Wang et al., 2021). Further, evaluating embedding methods on many tasks requires imple- menting multiple evaluation pipelines. Implemen- tation details like pre-processing or hyperparam- eters may influence the results making it unclear whether performance improvements simply come from a favorable evaluation pipeline. This leads to the “blind” application of these models to new use cases in industry or requires incremental work to reevaluate them on different tasks. The Massive Text Embedding Benchmark (MTEB) aims to provide clarity on how models perform on a variety of embedding tasks and thus serves as the gateway to finding universal text em- beddings applicable to a variety of tasks. MTEB consists of 58 datasets covering 112 languages from 8 embedding tasks: Bitext mining, classi- fication, clustering, pair classification, reranking, retrieval, STS and summarization. MTEB software is …

Content truncated to 500 words