PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models

PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models Eyal Ben-David∗ Carmel Rabinovitz∗ Roi Reichart Technion, Israel Institute of Technology {eyalbd12@campus.|carmelrab@campus.|roiri@}technion.ac.il Abstract Pivot-based neural representation models have led to significant progress in domain adapta- tion for NLP. However, previous research following this approach utilize only labeled data from the source domain and unlabeled data from the source and target domains, but neglect to incorporate massive unlabeled corpora that are not necessarily drawn from these domains. To alleviate this, we propose PERL: A representation learning model that extends contextualized word embedding mod- els such as BERT (Devlin et al., 2019) with pivot-based fine-tuning. PERL outperforms strong baselines across 22 sentiment classifica- tion domain adaptation setups, improves in- domain model performance, yields effective reduced-size models, and increases model stability.1 1 Introduction Natural Language Processing (NLP) algorithms are constantly improving, gradually approaching human-level performance (Dozat and Manning, 2017; Edunov et al., 2018; Radford et al., 2018). However, those algorithms often depend on the availability of large amounts of manually annota- ted data from the domain in which the task is performed. Unfortunately, collecting such anno- tated data is often costly and laborious, which substantially limits the applicability of NLP technology. Domain Adaptation (DA), training an algorithm on annotated data from a source domain so that it can be effectively applied to other target domains, is one of the ways to solve the above bottleneck. ∗Both authors contributed equally to this work. 1Our code is at https://github.com/eyalbd2/ PERL. Indeed, over the years substantialefforts have been devotedtothe DA challenge (Roark and Bacchiani, 2003; Daum´e III and Marcu, 2006; Ben-David et al., 2010; Jiang and Zhai, 2007; McClosky et al., 2010; Rush et al., 2012; Schnabel and Sch¨utze, 2014). Our focus in this paper is on unsupervised DA, the setup we consider most realistic. In this setup labeled data is available only from the source domain and unlabeled data is available from both the source and the target domains. While various approaches for DA have been proposed (§2), with the prominence of deep neural network (DNN) modeling, attention has been recently focused on representation learn- ing approaches. Within representation learning for unsupervised DA, two approaches have been shown particularly useful. In one line of work, DNN-based methods that use compress-based noise reduction to learn cross-domain features have been developed (Glorot et al., 2011; Chen et al., 2012). In another line of work, methods based on the distinction between pivot and non- pivot features (Blitzer et al., 2006, 2007) learn a joint feature representation for the source and the target domains. Later on, Ziser and Reichart (2017, 2018), and Li et al. (2018) married the two approaches and achieved substantial improve- ments on a variety of DA setups. Despite their success, pivot-based DNN models still only utilize labeled data from the source domain and unlabeled data from both the source and the target domains, but neglect to incorporate massive unlabeled corpora that are not necessarily drawn from these domains. With the recent game-changing success of contextualized …

Content truncated to 500 words