Neural Unsupervised Domain Adaptation in NLP—A Survey

Proceedings of the 28th International Conference on Computational Linguistics, pages 6838–6855 Barcelona, Spain (Online), December 8-13, 2020 6838 Neural Unsupervised Domain Adaptation in NLP—A Survey Alan Ramponi,1,2 Barbara Plank3 1Department of Inf. Eng. and Computer Science, University of Trento, Italy 2Fondazione The Microsoft Research – University of Trento Centre for Computational and Systems Biology (COSBI), Italy 3Department of Computer Science, ITU Copenhagen, Denmark ramponi@cosbi.eu, bplank@itu.dk Abstract Deep neural networks excel at learning from labeled data and achieve state-of-the-art results on a wide array of Natural Language Processing tasks. In contrast, learning from unlabeled data, especially under domain shift, remains a challenge. Motivated by the latest advances, in this survey we review neural unsupervised domain adaptation techniques which do not require labeled target domain data. This is a more challenging yet a more widely applicable setup. We outline methods, from early traditional non-neural methods to pre-trained model transfer. We also revisit the notion of domain, and we uncover a bias in the type of Natural Language Processing tasks which received most attention. Lastly, we outline future directions, particularly the broader need for out-of-distribution generalization of future NLP.1 1 Introduction Deep learning has undoubtedly pushed the frontier in Natural Language Processing (NLP). Particularly large pre-trained language models have improved results for a wide range of NLP applications. However, the lack of portability of NLP models to new conditions remains a central issue in NLP. For many target applications, labeled data is lacking (Y scarcity), and even for pre-training general models data might be scarce (X scarcity). This makes it even more pressing to revisit a particular type of transfer learning, namely domain adaptation (DA). A default assumption in many machine learning algorithms is that the training and test sets follow the same underlying distribution. When these distributions do not match, we face a dataset shift (Gretton et al., 2007) – in NLP typically referred to as a domain shift. In this setup, the target domain and the source training data differ, they are not sampled from the same underlying distribution. Consequently, performance drops on the target, which undermines the ability of models to truly generalize into the wild. Domain adaptation is closely tied to a fundamental bigger open issue in machine learning: generalization beyond the training distribution. Ultimately, intelligent systems should be able to adapt and robustly handle any test distribution, without having seen any data from it. This is the broader need for out-of-distribution generalization (Bengio, 2019), and a more challenging setup targeted at handling unknown domains (Volpi et al., 2018; Krueger et al., 2020). Work on domain adaptation focused largely on supervised domain adaptation (Daum´e III, 2007; Plank, 2011). In such a classic supervised DA setup, a small amount of labeled target domain data is available, along with some larger amount of labeled source domain data. The task is to adapt from the source to the specific target domain in light of limited target domain data. However, annotation is a substantial time- requiring and costly manual effort. While annotation directly …

Content truncated to 500 words