Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6716–6723 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6716 Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation Liyan Xu† Xuchao Zhang‡ Xujiang Zhao§ Haifeng Chen‡ Feng Chen§ Jinho D. Choi† †Emory University, {liyan.xu,jinho.choi}@emory.edu ‡NEC Laboratories America, {xuczhang,haifeng}@nec-labs.com §University of Texas at Dallas, {xujiang.zhao,feng.chen}@utdallas.edu Abstract Recent multilingual pre-trained language mod- els have achieved remarkable zero-shot perfor- mance, where the model is only finetuned on one source language and directly evaluated on target languages. In this work, we propose a self-learning framework that further utilizes unlabeled data of target languages, combined with uncertainty estimation in the process to select high-quality silver labels. Three dif- ferent uncertainties are adapted and analyzed specifically for the cross lingual transfer: Language Heteroscedastic/Homoscedastic Un- certainty (LEU/LOU), Evidential Uncertainty (EVI). We evaluate our framework with uncer- tainties on two cross-lingual tasks including Named Entity Recognition (NER) and Natu- ral Language Inference (NLI) covering 40 lan- guages in total, which outperforms the base- lines significantly by 10 F1 on average for NER and 2.5 accuracy score for NLI. 1 Introduction Recent multilingual pre-trained language models such as mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020) and mT5 (Xue et al., 2021) have demonstrated remarkable performance on var- ious direct zero-shot cross-lingual transfer tasks, where the model is finetuned on the source lan- guage, and directly evaluated on multiple target lan- guages that are unseen in the task-finetuning stage. While direct zero-shot transfer is a sensible testbed to assess the multilinguality of language models, one would apply supervised or semi-supervised learning on target languages to obtain more robust and accurate predictions in a practical scenario. In this work, we investigate self-learning (also known as “pseudo labels”) as one way to apply semi-supervised learning on cross-lingual transfer, where only unlabeled data of target languages are required, without any efforts to annotate gold la- bels for target languages. As self-learning has been proven effective in certain tasks of computer vision (Yalniz et al., 2019; Xie et al., 2020) and natural lan- guage processing (Artetxe et al., 2018; Dong and de Melo, 2019; Karan et al., 2020), we propose to formalize an iterative self-learning framework for multilingual tasks using pre-trained models, com- bined with explicit uncertainty estimation in the process to guide the cross-lingual transfer. Our self-learning (SL) framework utilizes any multilingual pre-trained models as the backbone, and iteratively grows the training set by adding pre- dictions of target language data as silver labels. We reckon two important observations from our prelim- inary study (baselines in §4). First, compared with self-training one target language at a time, jointly training multiple languages together can improve the performance on most languages, especially for certain low-resource languages that can achieve up to 8.6 F1 gain in NER evaluation. Therefore, our SL framework features the joint training strategy, maximizing potentials of different languages ben- efiting each other. Second, compared with simply using all unlabeled data as silver labels without considering prediction …

Content truncated to 500 words