Explicit Alignment Objectives for Multilingual Bidirectional Encoders

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3633–3643 June 6–11, 2021. ©2021 Association for Computational Linguistics 3633 Explicit Alignment Objectives for Multilingual Bidirectional Encoders Junjie Hu1∗, Melvin Johnson2, Orhan Firat2, Aditya Siddhant2, Graham Neubig1 1Carnegie Mellon University, 2Google Research {junjieh,gneubig}@cs.cmu.edu,{melvinp,orhanf,adisid}@google.com Abstract Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020a) have proven impres- sively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resource languages. This success comes despite the fact that there is no explicit ob- jective to align the contextual embeddings of words/sentences with similar meanings across languages together in the same space. In this paper, we present a new method for learn- ing multilingual encoders, AMBER (Aligned Multilingual Bidirectional EncodeR). AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different gran- ularities. We conduct experiments on zero- shot cross-lingual transfer learning for differ- ent tasks including sequence tagging, sentence retrieval and sentence classification. Experi- mental results on the tasks in the XTREME benchmark (Hu et al., 2020) show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLM-R-large model which has 3.2x the parameters of AM- BER. Our code and models are available at http://github.com/junjiehu/amber. 1 Introduction Cross-lingual embeddings, both traditional non- contextualized word embeddings (Faruqui and Dyer, 2014) and the more recent contextualized word embeddings (Devlin et al., 2019), are an es- sential tool for cross-lingual transfer in downstream applications. In particular, multilingual contextu- alized word representations have proven effective in reducing the amount of supervision needed in a variety of cross-lingual NLP tasks such as se- quence labeling (Pires et al., 2019), question an- swering (Artetxe et al., 2020), parsing (Wang et al., ∗*Work partially done at Google Research. 2019), sentence classification (Wu and Dredze, 2019) and retrieval (Yang et al., 2019a). Some attempts at training multilingual represen- tations (Devlin et al., 2019; Conneau et al., 2020a) simply train a (masked) language model on mono- lingual data from many languages. These methods can only implicitly learn which words and struc- tures correspond to each-other across languages in an entirely unsupervised fashion, but are nonethe- less quite effective empirically (Conneau et al., 2020b; K et al., 2020). On the other hand, some methods directly leverage multilingual parallel cor- pora (McCann et al., 2017; Eriguchi et al., 2018; Conneau and Lample, 2019; Huang et al., 2019; Siddhant et al., 2020), which gives some degree of supervision implicitly aligning the words in the two languages. However, the pressure on the model to learn clear correspondences between the con- textualized representations in the two languages is still implicit and somewhat weak. Because of this, several follow-up works (Schuster et al., 2019; Wang et al., 2020; Cao et al., 2020) have proposed methods that use word alignments from parallel corpora as the supervision signals …

Content truncated to 500 words