UDALM: Unsupervised Domain Adaptation through Language Modeling

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2579–2590 June 6–11, 2021. ©2021 Association for Computational Linguistics 2579 UDALM: Unsupervised Domain Adaptation through Language Modeling Constantinos Karouzos1, Georgios Paraskevopoulos1,4, Alexandros Potamianos1,2,3 1 School of ECE, National Technical University of Athens, Athens, Greece 2 Signal Analysis and Interpretation Laboratory (SAIL), USC, Los Angeles, CA, USA 3 Behavioral Signal Technologies, Los Angeles, CA, USA 4 Institute for Language and Speech Processing, Athena Research Center, Athens, Greece ckarouzos@gmail.com, geopar@central.ntua.gr, potam@central.ntua.gr Abstract In this work we explore Unsupervised Do- main Adaptation (UDA) of pretrained lan- guage models for downstream tasks. We intro- duce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target do- main distribution in a robust and sample effi- cient manner. Our experiments show that per- formance of models trained with the mixed loss scales with the amount of available tar- get data and the mixed loss can be effectively used as a stopping criterion during UDA train- ing. Furthermore, we discuss the relationship between A-distance and the target error and ex- plore some limitations of the Domain Adver- sarial Training approach. Our method is eval- uated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding 91.74% accuracy, which is an 1.11% absolute improve- ment over the state-of-the-art. 1 Introduction Deep architectures have achieved state-of-the-art results in a variety of machine learning tasks. How- ever, real world deployments of machine learning systems often operate under domain shift, which leads to performance degradation. This introduces the need for adaptation techniques, where a model is trained with data from a specific domain, and then can be optimized for use in new settings. Effi- cient techniques for model re-usability can lead to faster and cheaper development of machine learn- ing applications and facilitate their wider adoption. Especially techniques for Unsupervised Domain Adaptation (UDA) can have high real world im- pact, because they do not rely on expensive and time-consuming annotation processes to collect la- beled data for domain-specific supervised training, further streamlining the process. UDA approaches in the literature can be grouped in three major categories, namely pseudo-labeling techniques (e.g. Yarowsky, 1995; Zhou and Li, 2005), domain adversarial training (e.g. Ganin et al., 2016) and pivot-based approaches (e.g. Blitzer et al., 2006; Pan et al., 2010). Pseudo- labeling approaches use a model trained on the source labeled data to produce pseudo-labels for unlabeled target data and then train a model for the target domain in a supervised manner. Do- main adversarial training aims to learn a domain- independent mapping for input samples by adding an adversarial cost during model training, that min- imizes the distance between the source and target domain distributions. Pivot-based approaches aim to select domain-invariant features (pivots) and use them as a basis for cross-domain mapping. This work does not fall under any of these categories, rather we aim to optimize the fine-tuning procedure of pretrained language models …

Content truncated to 500 words