UXLA: A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLP

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1978–1992 August 1–6, 2021. ©2021 Association for Computational Linguistics 1978 UXLA: A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLP M Saiful Bari ∗ ¶, Tasnim Mohiuddin ∗¶, and Shafiq Joty¶§ ¶ Nanyang Technological University, Singapore § Salesforce Research Asia, Singapore {bari0001@e., mohi0004@e., srjoty@}ntu.edu.sg Abstract Transfer learning has yielded state-of-the-art (SoTA) results in many supervised NLP tasks. However, annotated data for every target task in every target language is rare, especially for low-resource languages. We propose UXLA a novel unsupervised data augmentation frame- work for zero-resource transfer learning sce- narios. In particular, UXLA aims to solve cross- lingual adaptation problems from a source language task distribution to an unknown target language task distribution, assuming no training label in the target language. At its core, UXLA performs simultaneous self- training with data augmentation and unsuper- vised sample selection. To show its effective- ness, we conduct extensive experiments on three diverse zero-resource cross-lingual trans- fer tasks. UXLA achieves SoTA results in all the tasks, outperforming the baselines by a good margin. With an in-depth framework dis- section, we demonstrate the cumulative contri- butions of different components to its success. 1 Introduction Self-supervised learning in the form of pretrained language models (LM) has been the driving force in developing state-of-the-art NLP systems in re- cent years. These methods typically follow two basic steps, where a supervised task-specific fine- tuning follows a large-scale LM pretraining (Rad- ford et al., 2019). However, getting labeled data for every target task in every target language is difficult, especially for low-resource languages. Recently, the pretrain-finetune paradigm has also been extended to multi-lingual setups to train effective multi-lingual models that can be used for zero-shot cross-lingual transfer. Jointly trained deep multi-lingual LMs like mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) coupled ∗Equal contribution with supervised fine-tuning in the source language have been quite successful in transferring linguistic and task knowledge from one language to another without using any task label in the target language. The joint pretraining with multiple languages al- lows these models to generalize across languages. Despite their effectiveness, recent studies (Pires et al., 2019; K et al., 2020) have also highlighted one crucial limiting factor for successful cross- lingual transfer. They all agree that the cross- lingual generalization ability of the model is limited by the (lack of) structural similarity between the source and target languages. For example, for trans- ferring mBERT from English, K et al. (2020) report about 23.6% accuracy drop in Hindi (structurally dissimilar) compared to 9% drop in Spanish (struc- turally similar) in cross-lingual natural language inference (XNLI). The difficulty level of transfer is further exacerbated if the (dissimilar) target lan- guage is low-resourced, as the joint pretraining step may not have seen many instances from this lan- guage in the first place. In our experiments (§3.2), in cross-lingual NER (XNER), we report F1 …

Content truncated to 500 words