Language-agnostic Representation from Multilingual Sentence Encoders for Cross-lingual Similarity Estimation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7764–7774 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7764 Language-Agnostic Representation from Multilingual Sentence Encoders for Cross-Lingual Similarity Estimation Nattapong Tiyajamorn1, Tomoyuki Kajiwara2, Yuki Arase3, Makoto Onizuka3 1 The University of Tokyo∗, nat-tiyajamorn@g.ecc.u-tokyo.ac.jp 2 Ehime University, kajiwara@cs.ehime-u.ac.jp 3 Osaka University, {arase, onizuka}@ist.osaka-u.ac.jp Abstract We propose a method to distil language- agnostic meaning embedding using a multilin- gual sentence encoder. By removing language- specific information from the original embed- ding, we retrieve an embedding that fully rep- resents the meaning of the sentence. The pro- posed method relies only on parallel corpora without any human annotations. Our meaning embedding allows for efficient cross-lingual sentence similarity estimation using a simple cosine similarity calculation. Experimental re- sults of both the quality estimation of machine translation and cross-lingual semantic textual similarity tasks reveal that our method con- sistently outperforms the strong baselines us- ing the original multilingual embeddings. The method also consistently improves the per- formance of any pre-trained multilingual sen- tence encoder, even in low-resource language pairs, where only tens of thousands of parallel sentence pairs are available.1 1 Introduction Pre-trained sentence encoders (Kiros et al., 2015; Logeswaran and Lee, 2018; Cer et al., 2018; De- vlin et al., 2019; Liu et al., 2019; Lan et al., 2020) boost the performance of various natural language understanding (NLU) tasks (Wang et al., 2018). Among them, the combination of self- attention networks (SANs) (Vaswani et al., 2017) and masked language modelling (Devlin et al., 2019; Liu et al., 2019) has proved to be remark- ably successful. These techniques are generalised across languages (K et al., 2020) and are even ap- plied to cross-lingual and multilingual NLU tasks such as cross-lingual semantic textual similarity (STS) (Cer et al., 2017) and quality estimation (QE) of machine translation (Specia et al., 2020). ∗This study was conducted when the first author was an undergraduate student at Osaka University. 1The source code for this paper is available at https: //github.com/nattaptiy/qe_disentangled zh en ne Figure 1: Visualisation of 1, 000 mBERT embeddings of parallel sentences in three languages: English (en), Chinese (zh), and Nepalese (ne) In the latest QE competitions at the confer- ence on machine translation (WMT) (Specia et al., 2020), all top-ranked systems (Ranasinghe et al., 2020; Fomicheva et al., 2020a; Nakamachi et al., 2020) employed pre-trained multilingual sentence encoders, such as multilingual BERT (mBERT) (Devlin et al., 2019) and XLM-RoBERTa (XLM-R) (Conneau and Lample, 2019; Conneau et al., 2020). These multilingual sentence encoders form a single self-attention network pre-trained on monolingual corpora in over 100 languages with the objective function of masked language mod- elling. Fine-tuning with a human-annotated corpus is mandatory for these models to enable them to estimate the semantic similarity between sentences across languages. Otherwise, these models are not sensitive to semantic similarity. A sentence encoder that can estimate semantic similarity across languages without fine-tuning for the target task is desirable because bilingual cor- pora with human annotations are unavailable in …

Content truncated to 500 words