Identifying Elements Essential for BERT’s Multilinguality

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4423–4437, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4423 Identifying Elements Essential for BERT’s Multilinguality Philipp Dufter, Hinrich Sch¨utze Center for Information and Language Processing (CIS), LMU Munich, Germany philipp@cis.lmu.de Abstract It has been shown that multilingual BERT (mBERT) yields high quality multilingual rep- resentations and enables effective zero-shot transfer. This is surprising given that mBERT does not use any crosslingual signal during training. While recent literature has studied this phenomenon, the reasons for the multilin- guality are still somewhat obscure. We aim to identify architectural properties of BERT and linguistic properties of languages that are necessary for BERT to become multilingual. To allow for fast experimentation we propose an efficient setup with small BERT models trained on a mix of synthetic and natural data. Overall, we identify four architectural and two linguistic elements that influence multilingual- ity. Based on our insights, we experiment with a multilingual pretraining setup that modifies the masking strategy using VecMap, i.e., unsu- pervised embedding alignment. Experiments on XNLI with three languages indicate that our findings transfer from our small setup to larger scale settings. 1 Introduction Multilingual models, i.e., models capable of pro- cessing more than one language with comparable performance, are central to natural language pro- cessing. They are useful as fewer models need to be maintained to serve many languages, resource re- quirements are reduced, and low- and mid-resource languages can benefit from crosslingual transfer. Further, multilingual models are useful in machine translation, zero-shot task transfer and typological research. There is a clear need for multilingual models for the world’s 7000+ languages. With the rise of static word embeddings, many multilingual embedding algorithms have been pro- posed (Mikolov et al., 2013; Hermann and Blun- som, 2014; Faruqui and Dyer, 2014); for a survey original (0) modified (8) modified +overparam (17) inv-order (3) 0.0 0.2 0.4 0.6 0.8 1.0 Performance Word Alignment Sent. Retrieval Word Translation Figure 1: Multilinguality in our BERT model (0) is harmed by three architectural modifications: lang-pos, shift-special, no-random (8); see §2.3 for definitions. Together with overparameterization almost no multilin- guality is left (17). Pairing a language with its inver- sion (i.e., inverted word order) destroys multilinguality as well (3). Having parallel training corpora is helpful for multilinguality (not shown). Results are for embed- dings from layer 8. see (Ruder et al., 2019). Pretrained language mod- els (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019) have high performance across tasks, outperforming static word embeddings. A simple multilingual model is multilingual BERT1 (mBERT). It is a BERT-Base model (Devlin et al., 2019) trained on the 104 largest Wikipedias with a shared subword vocabulary. There is no additional crosslingual signal. Still, mBERT yields high- quality multilingual representations (Pires et al., 2019; Wu and Dredze, 2019; Hu et al., 2020). The exact reason for mBERT’s multilinguality is – to the best of our knowledge – still debated. K et al. (2020) provide an extensive …

Content truncated to 500 words