Simplified Neural Unsupervised Domain Adaptation

Proceedings of NAACL-HLT 2019, pages 414–419 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 414 Simplified Neural Unsupervised Domain Adaptation Timothy Miller Computational Health Informatics Program Boston Children’s Hospital Harvard Medical School timothy.miller@childrens.harvard.edu Abstract Unsupervised domain adaptation (UDA) is the task of modifying a statistical model trained on labeled data from a source domain to achieve better performance on data from a target do- main, with access to only unlabeled data in the target domain. Existing state-of-the-art UDA approaches use neural networks to learn repre- sentations that can predict the values of subset of important features called “pivot features.” In this work, we show that it is possible to improve on these methods by jointly training the representation learner with the task learner, and examine the importance of existing pivot selection methods. 1 Introduction Unsupervised domain adaptation (UDA) is the task of modifying a statistical model trained on la- beled data from a source domain to achieve better performance on data from a target domain, with- out access to any labeled data in the target domain. Supervised domain adaptation methods can obtain excellent performance from a small number of la- beled examples in the target domain (Daum´e III, 2007), but UDA is attractive in cases where anno- tation requires specialized expertise or the num- ber of meaningfully different sub-domains is large (e.g., both are true for clinical NLP). Structural correspondence learning (Blitzer et al., 2006) (SCL) is one widely-used method for UDA in natural language processing. The key idea in SCL is that a subset of features, believed to be predictive across domains, are selected as pivot features. For each selected pivot feature, SCL creates an auxiliary classification task of pre- dicting the value of that feature in an instance, given the values of all the non-pivot features for that instance. The auxiliary classifiers therefore learn important cross-domain information about the structure of the feature space, which the SCL algorithm uses to create an augmented represen- tation that aligns features from different domains (further details in Section 2). Meanwhile, recent advances in neural network learning have shown that training regimens that jointly consider evidence from multiple sources can improve performance – both multi-task learn- ing (Søgaard and Goldberg, 2016) and fine tun- ing (Howard and Ruder, 2018; Devlin et al., 2018). However, existing SCL-based methods treat the representation learning and task learning as sep- arate tasks, so the parameters of the represen- tation learning machinery are fixed before train- ing for the downstream task. Jointly learning the representation- and task-specific parameters can potentially allow a learning algorithm to find rep- resentations that are better suited for the task. In this work, we describe a new UDA algorithm that is trained to jointly maximize two objectives: the primary supervised task in the source domain, and a pivot feature reconstruction task that can be trained on unlabeled data. We also explore the importance of pivot feature selection to this algorithm, in experiments that quantitatively and qualitatively examine …

Content truncated to 500 words