On the Language Neutrality of Pre-trained Multilingual Representations

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1663–1674 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1663 On the Language Neutrality of Pre-trained Multilingual Representations Jindˇrich Libovick´y1 and Rudolf Rosa2 and Alexander Fraser1 1Center for Information and Language Processing, LMU Munich, Germany 2Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic {libovicky, fraser}@cis.lmu.de rosa@ufal.mff.cuni.cz Abstract Multilingual contextual embeddings, such as multilingual BERT and XLM-RoBERTa, have proved useful for many multi-lingual tasks. Previous work probed the cross-linguality of the representations indirectly using zero-shot transfer learning on morphological and syntac- tic tasks. We instead investigate the language- neutrality of multilingual contextual embed- dings directly and with respect to lexical se- mantics. Our results show that contextual em- beddings are more language-neutral and, in general, more informative than aligned static word-type embeddings, which are explicitly trained for language neutrality. Contextual em- beddings are still only moderately language- neutral by default, so we propose two simple methods for achieving stronger language neu- trality: first, by unsupervised centering of the representation for each language and second, by fitting an explicit projection on small paral- lel data. Besides, we show how to reach state- of-the-art accuracy on language identification and match the performance of statistical meth- ods for word alignment of parallel sentences without using parallel data. 1 Introduction Multilingual BERT (mBERT; Devlin et al. 2019) gained popularity as a contextual representation for many multilingual tasks, e.g., dependency parsing (Kondratyuk and Straka, 2019a; Wang et al., 2019), cross-lingual natural language inference (XNLI) or named-entity recognition (NER) (Pires et al., 2019; Wu and Dredze, 2019; Kudugunta et al., 2019). Re- cently, a new pre-trained model, XLM-RoBERTa (XLM-R; Conneau et al. 2019), claimed to outper- form mBERT both on XNLI and NER tasks. We also study DistilBERT (Sanh et al., 2019) applied to mBERT, which promises to deliver comparable results to mBERT at a significantly lower computa- tional cost. Pires et al. (2019) present an exploratory paper showing that mBERT can be used cross-lingually for zero-shot transfer in morphological and syn- tactic tasks, at least for typologically similar lan- guages. They also study an interesting semantic task, sentence-retrieval, with promising initial re- sults. Their work leaves many open questions re- garding how well the cross-lingual mBERT rep- resentation captures lexical semantics, motivating our work. In this paper, we directly assess the cross-lingual properties of multilingual representations on tasks where lexical semantics plays an important role and present one unsuccessful and two successful methods for achieving better language neutrality. Multilingual capabilities of representations are often evaluated by zero-shot transfer from the train- ing language to a test language (Hu et al., 2020; Liang et al., 2020). However, in such a setup, we can never be sure if the probing model did not over- fit for the original language, as training is usually stopped when accuracy decreases on a validation set from the same language (otherwise, it would not be zero-shot), even when it would have been better to stop the training earlier. This …

Content truncated to 500 words