When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3610 - 3623 July 10-15, 2022 ©2022 Association for Computational Linguistics When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer Ameet Deshpande Department of Computer Science Princeton University, USA asd@cs.princeton.edu Partha Talukdar Google Research India partha@google.com Karthik Narasimhan Department of Computer Science Princeton University, USA karthikn@cs.princeton.edu Abstract While recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer, there is a lack of consensus in the community as to what shared properties between languages enable transfer on downstream tasks. Analyses in- volving pairs of natural languages are often in- conclusive and contradictory since languages simultaneously differ in many linguistic as- pects. In this paper, we perform a large-scale empirical study to isolate the effects of var- ious linguistic properties by measuring zero- shot transfer between four diverse natural lan- guages and their counterparts constructed by modifying aspects such as the script, word or- der, and syntax. Among other things, our ex- periments show that the absence of sub-word overlap significantly affects zero-shot transfer when languages differ in their word order, and there is a strong correlation between transfer performance and word embedding alignment between languages (e.g., ρs = 0.94 on the task of NLI). Our results call for focus in multilin- gual models on explicitly improving word em- bedding alignment between languages rather than relying on its implicit emergence.1 1 Introduction Multilingual language models like XLM (Con- neau et al., 2020a) and Multilingual-BERT (Devlin, 2019) are trained with masked-language modeling (MLM) objective on a combination of raw text from multiple languages. Surprisingly, these models ex- hibit decent cross-lingual zero-shot transfer, where fine-tuning on a task in a source language trans- lates to good performance for a different language (target). Requirements for zero-shot transfer Recent studies have provided inconsistent explanations for properties required for zero-shot transfer (hereon, 1Code is available at https://github.com/ princeton-nlp/MultilingualAnalysis transfer). For example, while Wu and Dredze (2019) conclude that sub-word overlap is vital for transfer, K et al. (2020) demonstrate that it is not crucial, although they consider only English as the source language. While Pires et al. (2019) suggest that typological similarity (e.g., similar SVO or- der) is essential for transfer, other works (Kakwani et al., 2020; Conneau et al., 2020a) successfully build multilingual models for dissimilar languages. Need for systematic analysis A major cause of these discrepancies is a large number of varying properties (e.g., syntax, script, and vocabulary size) between languages, which make isolating crucial ingredients for transfer difficult. Some studies al- leviate this issue by creating synthetic languages which differ from natural ones only in specific lin- guistic properties like script (K et al., 2020; Dufter and Schütze, 2020). However, their focus is only on English as a source language, and the scale of their experiments is small (in number of tasks or pre-training corpora size), thus limiting the scope of their findings to their settings alone. Our …

Content truncated to 500 words