Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1684–1697 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1684 Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training Kuan-Hao Huang, Wasi Uddin Ahmad, Nanyun Peng, Kai-Wei Chang University of California, Los Angeles {khhuang, wasiahmad, violetpeng, kwchang}@cs.ucla.edu Abstract Pre-trained multilingual language encoders, such as multilingual BERT and XLM-R, show great potential for zero-shot cross-lingual transfer. However, these multilingual encoders do not precisely align words and phrases across languages. Especially, learning align- ments in the multilingual embedding space usually requires sentence-level or word-level parallel corpora, which are expensive to be obtained for low-resource languages. An al- ternative is to make the multilingual encoders more robust; when fine-tuning the encoder us- ing downstream task, we train the encoder to tolerate noise in the contextual embedding spaces such that even if the representations of different languages are not aligned well, the model can still achieve good performance on zero-shot cross-lingual transfer. In this work, we propose a learning strategy for training ro- bust models by drawing connections between adversarial examples and the failure cases of zero-shot cross-lingual transfer. We adopt two widely used robust training methods, adver- sarial training and randomized smoothing, to train the desired robust model. The experi- mental results demonstrate that robust train- ing improves zero-shot cross-lingual transfer on text classification tasks. The improvement is more significant in the generalized cross- lingual transfer setting, where the pair of input sentences belong to two different languages. 1 Introduction Zero-shot cross-lingual transfer learning aims to learn models with data available in one or more source languages and use them in other target lan- guages for which there is no data (zero-resource) available. The zero-shot cross-lingual transfer has a great practical value for low-resource languages since it reduces the requirement of labeled data to learn models for downstream tasks, e.g., text clas- sification (Conneau et al., 2018; Yang et al., 2019) and question answering (Lewis et al., 2020). Recently, pre-trained multilingual language en- coders, such as multilingual BERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020a), demon- strate promising performance on zero-shot cross- lingual transfer learning for a wide range of down- stream tasks (Hu et al., 2020; Liang et al., 2020). These language encoders learn a shared multilin- gual contextual embedding space; they are able to represent word pairs in parallel sentences with similar contextual representations. However, the multilingual encoders fail to capture this similarity when the source and target languages are less simi- lar at levels of morphology, syntax, and semantics (Ahmad et al., 2019a,b). Prior studies (Cao et al., 2020; Pan et al., 2021; Dou and Neubig, 2021) have shown that aligning the representations of different languages in the multilingual embedding space plays an important role for zero-shot cross-lingual transfer learning. As illustrated in Figure 1a, words with similar meanings (e.g. this, ceci, and 这) have similar rep- resentations in the contextual multilingual embed- ding space, even though these words are in different languages. This alignment …

Content truncated to 500 words