Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 933–942 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 933 Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study Saurabh Kulshreshtha∗ Department of Computer Science University of Massachusetts Lowell skul@cs.uml.edu Jos´e Luis Redondo-Garc´ıa Amazon Alexa Cambridge, UK jluisred@amazon.com Ching-Yun Chang Amazon Alexa Cambridge, UK cychang@amazon.com Abstract Multilingual BERT (mBERT) has shown rea- sonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. Since mBERT is not pre-trained with ex- plicit cross-lingual supervision, transfer per- formance can further be improved by aligning mBERT with cross-lingual signal. Prior work proposes several approaches to align contex- tualised embeddings. In this paper we anal- yse how different forms of cross-lingual su- pervision and various alignment methods in- fluence the transfer capability of mBERT in zero-shot setting. Specifically, we compare parallel corpora vs. dictionary-based supervi- sion and rotational vs. fine-tuning based align- ment methods. We evaluate the performance of different alignment methodologies across eight languages on two tasks: Name Entity Recognition and Semantic Slot Filling. In addition, we propose a novel normalisation method which consistently improves the per- formance of rotation-based alignment includ- ing a notable 3% F1 improvement for distant and typologically dissimilar languages. Impor- tantly we identify the biases of the alignment methods to the type of task and proximity to the transfer language. We also find that super- vision from parallel corpus is generally supe- rior to dictionary alignments. 1 Introduction Multilingual BERT (mBERT) (Devlin et al., 2019) is the BERT architecture trained on data from 104 languages where all languages are embedded in the same vector space. Due to the multilingual and contextual representation properties of mBERT, it has gained popularity in various multilingual and cross-lingual tasks (Karthikeyan et al., 2020; Wu and Dredze, 2019). In particular, it has demon- strated good zero-shot cross-lingual transfer perfor- ∗ Work done during an internship at Amazon. mance on many downstream tasks, such as Docu- ment Classification, NLI, NER, POS tagging, and Dependency Parsing (Wu and Dredze, 2019), when the source and the target languages are similar. Many experiments (Ahmad et al., 2019) suggest that to achieve reasonable performance in the zero- shot setup, the source and the target languages need to share similar grammatical structure or lie in the same language family. In addition, since mBERT is not trained with explicit language signal, mBERT’s multilingual representations are less effective for languages with little lexical overlap (Patra et al., 2019). One branch of work is therefore dedicated to improve the multilingual properties of mBERT by aligning the embeddings of different languages with cross-lingual supervision. Broadly, two methods have been proposed in prior work to induce cross-lingual signals in con- textual embeddings: 1) Rotation Alignment as de- scribed in Section 2 aims at learning a linear rota- tion transformation to project source language em- beddings into their respective locations in the target language space (Schuster et al., 2019b; Wang et al., 2019; Aldarmaki and Diab, 2019); 2) Fine-tuning Alignment as explained in …

Content truncated to 500 words