Margin-aware Unsupervised Domain Adaptation for Cross-lingual Text Labeling

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3527–3536 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3527 Margin-aware Unsupervised Domain Adaptation for Cross-lingual Text Labeling Dejiao Zhang1 Ramesh Nallapati1 Henghui Zhu1 Feng Nan1 Cicero Nogueira dos Santos1 Kathleen McKeown1,2 Bing Xiang1 1AWS AI 2 Department of Computer Science, Columbia University dejiaoz,henghui,nanfen,cicnog,rnallapa,mckeownk,bxiang@amazon.com Abstract Unsupervised domain adaptation addresses the problem of leveraging labeled data in a source domain to learn a well-performing model in a target domain where labels are unavailable. In this paper, we improve upon a recent theoreti- cal work (Zhang et al., 2019b) and adopt the Margin Disparity Discrepancy (MDD) unsu- pervised domain adaptation algorithm to solve the cross-lingual text labeling problems. Ex- periments on cross-lingual document classifi- cation and NER demonstrate the proposed do- main adaptation approach advances the state- of-the-art results by a large margin. Specif- ically, we improve MDD by efficiently opti- mizing the margin loss on the source domain via Virtual Adversarial Training (VAT). This bridges the gap between theory and the loss function used in the original work Zhang et al. (2019b), and thereby significantly boosts the performance. Our numerical results also indi- cate that VAT can remarkably improve the gen- eralization performance of both domains for various domain adaptation approaches. 1 Introduction Unsupervised domain adaptation provides an ap- pealing solution to many applications where di- rect access to a massive amount of labeled data is prohibitive or very costly (Sun and Saenko, 2014; Vazquez et al., 2013; Stark et al., 2010; Keung et al., 2019). For example, we often have suffi- cient labeled data for English, while very limited or even no labeled data are available for many other languages. Successfully transferring knowledge learned from the English domain to other languages is of great interest in solving many tasks in natural language processing. Many recent successes in unsupervised domain adaptation have been achieved by learning domain invariant features that are simultaneously being dis- criminative to the task in the source domain (Chen et al., 2018; Ganin and Lempitsky, 2014; Ganin et al., 2016; Tzeng et al., 2017). Following this line, Keung et al. (2019) propose a language-adversarial training approach for cross-lingual document clas- sification and NER. They leverage the benefit of contextualized word embeddings by using multi- lingual BERT (Devlin et al., 2019) as the feature generator, and adopt the GAN framework (Goodfel- low et al., 2014) to align the features from the two domains. Keung et al. (2019) show significant im- provement over the baseline where the pretrained multilingual BERT is finetuned on the English data alone and testing on the same tasks in other lan- guages. However, Keung et al. (2019), as well as the works mentioned above, are inspired by the pioneering work of Ben-David et al. (2010), which only rigorously studies domain adaptation in the setting of binary classification; there is a lack of theoretical guarantees when it comes to multiclass classification. In this work, we are instead motivated by a recent work (Zhang …

Content truncated to 500 words