Supervised Contrastive Learning for Cross-lingual Transfer Learning

Supervised Contrastive Learning for Cross-lingual Transfer Learning Shuaibo Wang1, Hui Di2, Hui Huang3, Siyu Lai1 Kazushige Ouchi2, Yufeng Chen1∗, Jinan Xu1 1 School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China 2 Toshiba (China) Co., Ltd. 3 Harbin Institute of Technology {wangshuaibo,20120374,chenyf,jaxu}@bjtu.edu.cn; dihui@toshiba.com.cn; huanghui hit@126.com; kazushige.ouchi@toshiba.co.jp Abstract Multilingual pre-trained representations are not well-aligned by nature, which harms their per- formance on cross-lingual tasks. Previous methods propose to post-align the multilingual pre- trained representations by multi-view alignment or contrastive learning. However, we argue that both methods are not suitable for the cross-lingual classification objective, and in this paper we propose a simple yet effective method to better align the pre-trained representations. On the basis of cross-lingual data augmentations, we make a minor modification to the canonical contrastive loss, to remove false-negative examples which should not be contrasted. Augmentations with the same class are brought close to the anchor sample, and augmentations with different class are pushed apart. Experiment results on three cross-lingual tasks from XTREME benchmark show our method could improve the transfer performance by a large margin with no additional resource needed. We also provide in-detail analysis and comparison between different post-alignment strategies. 1 Introduction Cross-lingual transfer learning aims to transfer the learned knowledge from a resource-rich language to a resource-lean language. The main idea of crosss-lingual transfer is to learn a shared language-invariant feature space for both languages, so that a model trained on the source language could be applied to the target language directly. Such generalization ability greatly reduces the required annotation efforts, and has urgent demand in real-world applications. Recent multilingual pre-trained models, such as XLM-RoBERTa(XLM-R) (Conneau et al., 2020), have been demonstrated surprisingly effective in the cross-lingual scenario. By fine-tuning on labeled data in a source language, such models can generalize to other target languages even without any addi- tional training. This has become a de-facto paradigm for cross-lingual language understanding tasks. Despite their success in cross-lingual transfer tasks, multilingual pre-training commonly lacks explicit cross-lingual supervision, and the representations for different languages are not inherently aligned. To further improve the transferability of multilingual pre-trained representations, previous works propose different methods for cross-lingual alignment. Zheng et al. (2021) and Lai et al. (2021) propose to augment the training set with different views, and align the pre-trained representations of different lan- guages by dragging two views closer. However, simply bringing different views closer would easily lead to representation collapse and performance degradation (Tao et al., 2021). Meanwhile, Pan et al. (2021) and Wei et al. (2021) propose to incorporate additional parallel data, and align the pre-trained representa- tions by contrasting positive and negative samples. However, monotonously treating all random samples equally negative is inconsistent with the classification objective. In this work, we propose a simple yet effective method to better post-align the multilingual repre- sentations on downstream tasks, which can both avoid representation collapse and meanwhile induce classification bias. With only training data for the source language available, our method performs cross- lingual fine-tuning by …

Content truncated to 500 words