Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4702 - 4712 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure Yuan Chai1,∗, Yaobo Liang2, Nan Duan2 1 Beihang University 2 Microsoft Research Asia 1chaiyuan@buaa.edu.cn 2{yalia, nanduan}@microsoft.com Abstract Multilingual pre-trained language models, such as mBERT and XLM-R, have shown impres- sive cross-lingual ability. Surprisingly, both of them use multilingual masked language model (MLM) without any cross-lingual supervision or aligned data. Despite the encouraging re- sults, we still lack a clear understanding of why cross-lingual ability could emerge from multi- lingual MLM. In our work, we argue that cross- language ability comes from the commonal- ity between languages. Specifically, we study three language properties: constituent order, composition and word co-occurrence. First, we create an artificial language by modifying property in source language. Then we study the contribution of modified property through the change of cross-language transfer results on target language. We conduct experiments on six languages and two cross-lingual NLP tasks (textual entailment, sentence retrieval). Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer. 1 Introduction Zero-Shot Cross-Lingual transfer aims to build models for the target language by reusing knowl- edge learned from the source language. In this way, models can be efficiently implemented in multilin- gual as well as low-resource language scenarios. Traditionally, it is solved by a two-step pipeline (Ruder et al., 2019): a shared multilingual textual representation is first built and then supervised data from the source language is used on the top of it to train task-specific models. With the recent emergence of multilingual language models, the standard paradigm in this field has shifted to the pre-trained fine-tuning paradigm. Multilingual pre- trained language models, such as mBERT (Devlin ∗ Work done during internship at Microsoft Research Asia. I 我 在 读 两篇 论文 。 read two papers Saturday . ROOT S . . VP NP VBD NP PRP read CD NNS PP papers two I IN ROOT IP PU 。 VP VP PP NP PN VV NP 读 我 在 P NP 周六 AS 了 两篇 QP NP 论文 English: Chinese: 周六 on NP Saturday on 了 Figure 1: Example of two sentences in different lan- guages, which are different in constituent order, but are very similar in constituent tree. Note that we simplify the constituent tree for better understanding. et al., 2019) and XLM-R (Conneau et al., 2020a), have proven effective for cross-lingual transfer with better results on a large number of downstream tasks and languages (Pires et al., 2019; Conneau et al., 2020a). The most surprising part is that mBERT and XLM-R are both trained without using any parallel corpus. Previous work (Pires et al., 2019; Wu and Dredze, 2019) attribute this success to the shared anchor words. But recent work (Conneau et al., 2020b; K et al., …

Content truncated to 500 words