Unsupervised Domain Adaptation for Text Classification via Meta Self-Paced Learning

Proceedings of the 29th International Conference on Computational Linguistics, pages 4741–4752 October 12–17, 2022. 4741 Unsupervised Domain Adaptation for Text Classification via Meta Self-Paced Learning Nghia Ngo Trung1, Linh Ngo Van2 and Thien Huu Nguyen1 1 Department of Computer and Information Science, University of Oregon, Eugene, OR, USA 2 Hanoi University of Science and Technology, Vietnam {nghian@,thien@cs}.uoregon.edu, linhnv@soict.hust.edu.vn Abstract A shift in data distribution can have a signifi- cant impact on performance of a text classifi- cation model. Recent methods addressing un- supervised domain adaptation for textual tasks typically extracted domain-invariant represen- tations through balancing between multiple ob- jectives to align feature spaces between source and target domains. While effective, these methods induce various new domain-sensitive hyperparameters, thus are impractical as large- scale language models are drastically growing bigger to achieve optimal performance. To this end, we propose to leverage meta-learning framework to train a neural network-based self-paced learning procedure in an end-to- end manner. Our method, called Meta Self- Paced Domain Adaption (MSP-DA), follows a novel but intuitive domain-shift variation of cluster assumption to derive the meta train-test dataset split based on the self-pacing difficul- ties of source domain’s examples. As a result, MSP-DA effectively leverages self-training and self-tuning domain-specific hyperparameters simultaneously throughout the learning pro- cess. Extensive experiments demonstrate our framework substantially improves performance on target domains, surpassing state-of-the-art approaches. Detailed analyses validate our method and provide insight into how each do- main affects the learned hyperparameters. 1 Introduction Given enough supervision, modern deep learning models can learn a new task with great accuracy. However, in many practical settings, the goal is to adapt to a new domain in which there is a differ- ent in data distribution between training and test- ing processes. This poses a major challenge for standard natural language systems due to both the intrinsic variation of linguistics (e.g., lexical shift, semantic shift) as well as the extrinsic factors such as how textual datasets are collected and annotated. For example, a model trained to predict news events Figure 1: An example where domain shift between source domain (grey colors) and target domain (deep color) results in significant overlaps between high-loss regions of source decision boundary (lime) with high-density target clusters. may easily recognize, from medical domain, "died" as an event, but would not be able to detect obvi- ous events such as "mutation" or "cancer". Such a model may even fail to generalize to closer adap- tation settings (e.g. news from different times and sources). The majority of existing unsupervised domain adaptatiopn (UDA) approaches combined various training objectives to align different aspects of domain-specific extracted features. In particular, the most prominent approach is domain-adversarial neural network (DANN) (Ganin et al., 2016) that employs a domain-adversarial training procedure between a domain classifier and the network’s fea- ture extractor to learn a discriminative and domain- invariant joint feature representation. The sim- plicity of DANN allows researchers to incorpo- rate it with multiple other objectives such as semi- supervised learning (SSL) regularizers (Shu et al., 2018), discrepancy …

Content truncated to 500 words