Unsupervised Energy-based Adversarial Domain Adaptation for Cross-domain Text Classification

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1208–1218 August 1–6, 2021. ©2021 Association for Computational Linguistics 1208 Unsupervised Energy-based Adversarial Domain Adaptation for Cross-domain Text Classification Han Zou∗ Microsoft hanzou@microsoft.com Jianfei Yang∗ Nanyang Technological University yang0478@ntu.edu.sg Xiaojian Wu† Microsoft xiaojian.academy@gmail.com Abstract Transferring knowledge from a label-rich do- main (source domain) to a label-scarce domain (target domain) for pervasive cross-domain Text Classification (TC) is a non-trivial task. To overcome this issue, we propose EADA, a novel unsupervised energy-based adversarial domain adaptation framework. First, a deep pre-trained language model (e.g. RoBERTa) is leveraged as a shared feature extractor that maps the text sequences from both source and target domains to a feature space. Since the source features maintain good feature discrim- inability because of the full supervised train- ing, we design a method that encourages target features towards the source ones via adversar- ial learning. An autoencoder is designed as an energy function that focuses on reconstructing source feature embeddings, while the feature extractor aims to generate source-like target feature embeddings to deceive the autoencoder. In this manner, the target feature embeddings become domain-invariant and inherit great dis- criminability. Extensive experiments on multi- domain sentiment classification (Amazon re- view dataset) and Yes/No question-answering classification (BoolQ and MARCO dataset) are conducted. The experimental results val- idate that EADA largely alleviates the do- main discrepancy while maintaining excellent discriminability and achieves state-of-the-art cross-domain TC performance. 1 Introduction With the booming development of Natural Lan- guage Processing (NLP) in recent years, text clas- sification (TC) is playing a vital role in a myriad of services in our daily lives, such as online rec- ommendations, email spam detection, sentiment classification and social media analysis. Large pre- trained language models, e.g. BERT (Devlin et al., ∗Equal Contribution. †Work done while at Microsoft. 2019), XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019b), achieve outstanding results on challenging NLP benchmarks, i.e. GLUE (Wang et al., 2018), RACE (Lai et al., 2017), and SQuAD (Rajpurkar et al., 2016). These models enable nu- merous downstream NLP tasks with compelling performance, including TC, where the model is further fine-tuned with annotated data. TC tasks are usually domain dependent in real- world. Thus, the performance of these powerful deep models is still fluctuated and even degraded when directly implementing them in a unseen do- main (target domain), where the task topic or the data distributions are different from the domain during training (source domain). Although their performance can be improved via fine-tuning with full supervision in the target domain, a significant amount of labeled target data is required. Collect- ing high-quality data is usually difficult and expen- sive in many real-world domains. Furthermore, the annotating process is extremely time-consuming and labor-intensive. To overcome these issues, un- supervised domain adaptation (UDA), which aims to transfer the knowledge from a label-rich domain (source domain) to a label-scarce or unlabeled do- main (target domain) is proposed (Li et al., 2017; Chen et al., 2018; Guo et al., 2018; Zhang et …

Content truncated to 500 words