Consistency Regularization for Cross-Lingual Fine-Tuning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3403–3417 August 1–6, 2021. ©2021 Association for Computational Linguistics 3403 Consistency Regularization for Cross-Lingual Fine-Tuning Bo Zheng†∗, Li Dong‡, Shaohan Huang‡, Wenhui Wang‡, Zewen Chi‡∗ Saksham Singhal‡, Wanxiang Che†, Ting Liu†, Xia Song‡, Furu Wei‡ †Harbin Institute of Technology ‡Microsoft Corporation {bzheng,car,tliu}@ir.hit.edu.cn {lidong1,shaohanh,wenwan,saksingh,xiaso,fuwei}@microsoft.com Abstract Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others. In this work, we propose to improve cross-lingual fine- tuning with consistency regularization. Specif- ically, we use example consistency regulariza- tion to penalize the prediction sensitivity to four types of data augmentations, i.e., sub- word sampling, Gaussian noise, code-switch substitution, and machine translation. In ad- dition, we employ model consistency to regu- larize the models trained with two augmented versions of the same training set. Experimen- tal results on the XTREME benchmark show that our method1 significantly improves cross- lingual fine-tuning across various tasks, includ- ing text classification, question answering, and sequence labeling. 1 Introduction Pre-trained cross-lingual language models (Con- neau and Lample, 2019; Conneau et al., 2020a; Chi et al., 2020) have shown great transferability across languages. By fine-tuning on labeled data in a source language, the models can generalize to other target languages, even without any additional training. Such generalization ability reduces the required annotation efforts, which is prohibitively expensive for low-resource languages. Recent work has demonstrated that data aug- mentation is helpful for cross-lingual transfer, e.g., translating source language training data into target languages (Singh et al., 2019), and generating code- switch data by randomly replacing input words in the source language with translated words in tar- get languages (Qin et al., 2020). By populating the dataset, their fine-tuning still treats training ∗Contribution during internship at Microsoft Research. 1The code is available at https://github.com/ bozheng-hit/xTune. instances independently, without considering the inherent correlations between the original input and its augmented example. In contrast, we propose to utilize consistency regularization to better leverage data augmentation for cross-lingual fine-tuning. In- tuitively, for a semantic-preserving augmentation strategy, the predicted result of the original input should be similar to its augmented one. For ex- ample, the classification predictions of an English sentence and its translation tend to remain consis- tent. In this work, we introduce a cross-lingual fine- tuning method XTUNE that is enhanced by con- sistency regularization and data augmentation. First, example consistency regularization enforces the model predictions to be more consistent for semantic-preserving augmentations. The regular- izer penalizes the model sensitivity to different sur- face forms of the same example (e.g., texts written in different languages), which implicitly encour- ages cross-lingual transferability. Second, we in- troduce model consistency to regularize the mod- els trained with various augmentation strategies. Specifically, given two augmented versions of the same training set, we encourage the models trained on these two datasets to make consistent predic- tions for the same example. The method enforces the corpus-level consistency between the distribu- tions learned by …

Content truncated to 500 words