A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning

Proceedings of the 29th International Conference on Computational Linguistics, pages 4372–4380 October 12–17, 2022. 4372 A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning Kunbo Ding1 ∗, Weijie Liu1,2 †, Yuejian Fang1, Weiquan Mao2, Zhe Zhao2 Tao Zhu2, Haoyan Liu2, Rong Tian2, Yiren Chen2 1Peking University, Beijing, China 2Tencent Research, Beijing, China kunbo_ding@stu.pku.edu.cn, dataliu@pku.edu.cn, fangyj@ss.pku.edu.cn {weiquanmao, nlpzhezhao, mardozhu, haoyanliu, rometian, yirenchen}@tencent.com Abstract Existing zero-shot cross-lingual transfer meth- ods rely on parallel corpora or bilingual dic- tionaries, which are expensive and impractical for low-resource languages. To disengage from these dependencies, researchers have explored training multilingual models on English-only resources and transferring them to low-resource languages. However, its effect is limited by the gap between embedding clusters of different languages. To address this issue, we propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to vir- tual multilingual embeddings without semantic loss, thereby improving cross-lingual transfer- ability. Experimental results on mBERT and XLM-R demonstrate that our method signifi- cantly outperforms previous works on the zero- shot cross-lingual text classification task and can obtain a better multilingual alignment. 1 Introduction In recent years, advances in multilingual models such as mBERT (Devlin et al., 2019), XLM (Con- neau and Lample, 2019), XLM-R (Conneau et al., 2020), etc., after being fine-tuned with annotated data, have enabled significant improvements in many cross-lingual tasks. However, due to the lack of annotated data, some tasks in low-resource languages have not enjoyed this technological ad- vancement. To solve this issue, the academic and industrial community began to focus on zero- shot cross-lingual transfer learning (Huang et al., 2019; Artetxe et al., 2020), which aims to fine-tune multilingual models with annotated data in high- resource languages and obtain a nice performance in low-resource language tasks. Some works aligned word embeddings between high- and low-resource languages through addi- tional parallel sentence pairs (Artetxe and Schwenk, ∗ Contribution during internship at Tencent Inc. † Corresponding author: Weijie Liu. (b) mBERTbase (Cao et al.) (a) mBERTbase (Libovicky et al.) English Bavarian Low Saxon Japanese West Frisian Dutch Spanish Scots French German Australian Chinese … … … … Irish (d) Ours (c) Zero-shot with Smoothing (Huang et al.) processing language natural 处理 语言 自然 próiseáil teanga nádúrtha English Synonyms for “natural” English Synonyms for “language” English Synonyms for “processing” Figure 1: (a) Different languages clusters in mBERT. (b) The relative positions of "nature", "language" and "processing" are similar in English, Chinese and Irish (Cao et al., 2020). (c) Using synonym augmentation to train a robust region covering words in other languages. (d) We align different languages and construct a suitable robust region by pushing the embeddings away and pulling the relative distance among words. 2019; Wei et al., 2021; Chi et al., 2021; Pan et al., 2021) or bilingual dictionaries (Cao et al., 2020; Qin et al., 2020; Liu et al., 2020), so that high- resource fine-tuned models can be transferred to low-resource languages. Although this approach has achieved excellent results in many languages, parallel corpora and bilingual dictionaries are still …

Content truncated to 500 words