Language-agnostic BERT Sentence Embedding

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 878 - 891 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Language-agnostic BERT Sentence Embedding Fangxiaoyu Feng∗, Yinfei Yang∗†, Daniel Cer, Naveen Arivazhagan, Wei Wang† Google AI Mountain View {fangxiaoyu, cer, navari}@google.com {yangyin7, wei.wang.world}@gmail.com Abstract While BERT is an effective method for learn- ing monolingual sentence embeddings for se- mantic similarity and embedding based trans- fer learning (Reimers and Gurevych, 2019), BERT based cross-lingual sentence embed- dings have yet to be explored. We sys- tematically investigate methods for learning multilingual sentence embeddings by combin- ing the best methods for learning monolin- gual and cross-lingual representations includ- ing: masked language modeling (MLM), trans- lation language modeling (TLM) (Conneau and Lample, 2019), dual encoder translation ranking (Guo et al., 2018), and additive margin softmax (Yang et al., 2019a). We show that in- troducing a pre-trained multilingual language model dramatically reduces the amount of par- allel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 lan- guages on Tatoeba, well above the 65.5% achieved by Artetxe and Schwenk (2019b), while still performing competitively on mono- lingual transfer learning benchmarks (Con- neau and Kiela, 2018). Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/ google/LaBSE. 1 Introduction In this paper, we systematically explore using pre- training language models in combination with the best of existing methods for learning cross-lingual sentence embeddings. Such embeddings are use- ful for clustering, retrieval, and modular use of text representations for downstream tasks. While ∗Equal contributions. †Work done while at Google. Pre-trained BERT 12-Layer Transformer Embedding Network Source Text Target Text Share Parameters Loss Add Additive Margin Initialize Initialize Source Embeddings Target Embeddings 12-Layer Transformer Embedding Network Figure 1: Dual encoder model with BERT based encod- ing modules. existing cross-lingual sentence embedding mod- els incorporate large transformer models, using large pretrained language models is not well ex- plored. Rather in prior work, encoders are trained directly on translation pairs (Artetxe and Schwenk, 2019b; Guo et al., 2018; Yang et al., 2019a), or on translation pairs combined with monolingual input- response prediction (Chidambaram et al., 2019; Yang et al., 2019b). In our exploration, as illustrated in figure 1, we make use of dual-encoder models, which have been demonstrated as an effective approach for learning bilingual sentence embeddings (Guo et al., 2018; Yang et al., 2019a). However, diverging from prior work, rather than training encoders from scratch, we investigate using pre-trained encoders based on large language models. We contrast models with and without additive margin softmax (Yang et al., 2019a)1. Figure 2 illustrates where our work stands (shaded) in the field of LM pre-training and sentence embedding learning. Our massively multilingual models outperform the previous state-of-the-art on large bi-text re- trieval tasks …

Content truncated to 500 words