LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1886–1894 May 2-6, 2023 ©2023 Association for Computational Linguistics LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation Zhuoyuan Mao∗and Tetsuji Nakagawa Google Research kevinmzy@gmail.com, tnaka@google.com Abstract Large-scale language-agnostic sentence em- bedding models such as LaBSE (Feng et al., 2022) obtain state-of-the-art perfor- mance for parallel sentence alignment. How- ever, these large-scale models can suffer from inference speed and computation over- head. This study systematically explores learning language-agnostic sentence embed- dings with lightweight models. We demon- strate that a thin-deep encoder can construct robust low-dimensional sentence embeddings for 109 languages. With our proposed distil- lation methods, we achieve further improve- ments by incorporating knowledge from a teacher model. Empirical results on Tatoeba, United Nations, and BUCC show the effec- tiveness of our lightweight models. We re- lease our lightweight language-agnostic sen- tence embedding models LEALLA on Tensor- Flow Hub.1 1 Introduction Language-agnostic sentence embedding mod- els (Artetxe and Schwenk, 2019b; Yang et al., 2020; Reimers and Gurevych, 2020; Feng et al., 2022; Mao et al., 2022) align multiple languages in a shared embedding space, facilitating parallel sen- tence alignment that extracts parallel sentences for training translation systems (Schwenk et al., 2021). Among them, LaBSE (Feng et al., 2022) achieves the state-of-the-art parallel sentence alignment ac- curacy over 109 languages. However, 471M pa- rameters of LaBSE lead to the computationally- heavy inference. The 768-dimensional sentence embeddings of LaBSE (LaBSE embeddings) make it suffer from computation overhead of downstream tasks (e.g., kNN search). This limits its applica- tion on resource-constrained devices. Therefore, ∗Currently at Kurohashi-Chu-Murawaki Lab., Kyoto Uni- versity. Work done during Google internship. 1https://tfhub.dev/s?q=LEALLA we explore training a lightweight model to gener- ate low-dimensional sentence embeddings while retaining the performance of LaBSE. We first investigate the performance of dimension-reduced LaBSE embeddings and show that it performs comparably with LaBSE. Subse- quently, we experiment with various architectures to see whether such effective low-dimensional em- beddings can be obtained from a lightweight en- coder. We observe that the thin-deep (Romero et al., 2015) architecture is empirically superior for learn- ing language-agnostic sentence embeddings. Di- verging from previous work, we show that low- dimensional embeddings based on a lightweight model are effective for parallel sentence alignment of 109 languages. LaBSE benefits from multilingual language model pre-training, but no multilingual pre-trained models are available for the lightweight architec- tures. Thus, we propose two knowledge distillation methods to further enhance the lightweight models by forcing the model to extract helpful information from LaBSE. We present three lightweight mod- els improved with distillation: LEALLA-small, LEALLA-base, and LEALLA-large, with 69M, 107M, and 147M parameters, respectively. Fewer model parameters and their 128-d, 192-d, and 256- d sentence embeddings are expected to accelerate downstream tasks, while the performance drop of merely up to 3.0, 1.3, and 0.3 P@1 (or F1) points is observed on three benchmarks of parallel sentence alignment. In addition, we show the effectiveness of each loss function through an ablation study. …

Content truncated to 500 words