First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2214–2231 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2214 First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT Benjamin Muller1,2 Yanai Elazar3,4 Benoˆıt Sagot1 Djamé Seddah1 1Inria, Paris, France 2Sorbonne Université, Paris, France 3Computer Science Department, Bar Ilan University 4Allen Institute for Artificial Intelligence {benjamin.muller, benoit.sagot,djame.seddah}@inria.fr yanaiela@gmail.com Abstract Multilingual pretrained language models have demonstrated remarkable zero-shot cross- lingual transfer capabilities. Such transfer emerges by fine-tuning on a task of interest in one language and evaluating on a distinct lan- guage, not seen during the fine-tuning. De- spite promising results, we still lack a proper understanding of the source of this transfer. Using a novel layer ablation technique and analyses of the model’s internal representa- tions, we show that multilingual BERT, a popular multilingual language model, can be viewed as the stacking of two sub-networks: a multilingual encoder followed by a task- specific language-agnostic predictor. While the encoder is crucial for cross-lingual trans- fer and remains mostly unchanged during fine- tuning, the task predictor has little importance on the transfer and can be reinitialized during fine-tuning. We present extensive experiments with three distinct tasks, seventeen typologi- cally diverse languages and multiple domains to support our hypothesis. 1 Introduction Zero-shot Cross-Lingual transfer aims at building models for a target language by reusing knowledge acquired from a source language. Historically, it has been tackled with a two-step standard cross- lingual pipeline (Ruder et al., 2019): (1) Building a shared multilingual representation of text, typ- ically by aligning textual representations across languages. This step can be done using feature extraction (Aone and McKee, 1993; Schultz and Waibel, 2001) as with the delexicalized approach (Zeman and Resnik, 2008; Søgaard, 2011) or using word embedding techniques (Mikolov et al., 2013; Smith et al., 2017) by projecting monolingual em- beddings onto a shared multilingual embedding space, this step requiring explicit supervision sig- nal in the target language in the form of features or parallel data. (2) Training a task-specific model using supervision on a source language on top of the shared representation. Recently, the rise of multilingual language mod- els entailed a paradigm shift in this field. Multi- lingual pretrained language models (Devlin et al., 2019; Conneau and Lample, 2019) have been shown to perform efficient zero-shot cross-lingual transfer for many tasks and languages (Pires et al., 2019; Wu and Dredze, 2019). Such transfer re- lies on three-steps: (i) pretraining a mask-language model (e.g. Devlin et al. (2019)) on the concatena- tion of monolingual corpora across multiple lan- guages, (ii) fine-tuning the model on a specific task in the source language, and (iii) using the fine- tuned model on a target language. The success of this approach is remarkable, and in contrast to the standard cross-lingual pipeline, the model sees nei- ther aligned data nor task-specific annotated data in the target language at any training stage. The source of such a successful transfer is still largely …

Content truncated to 500 words