Pivot Based Language Modeling for Improved Neural Domain Adaptation

Proceedings of NAACL-HLT 2018, pages 1241–1251 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Pivot Based Language Modeling for Improved Neural Domain Adaptation Yftah Ziser and Roi Reichart Faculty of Industrial Engineering and Management, Technion, IIT syftah@campus.technion.ac.il, roiri@ie.technion.ac.il Abstract Representation learning with pivot-based methods and with Neural Networks (NNs) have lead to significant progress in domain adaptation for Natural Language Processing. However, most previous work that follows these approaches does not explicitly exploit the structure of the input text, and its output is most often a single representation vector for the entire text. In this paper we present the Pivot Based Language Model (PBLM), a representation learning model that marries together pivot-based and NN modeling in a structure aware manner. Particularly, our model processes the information in the text with a sequential NN (LSTM) and its output consists of a context-dependent representation vector for every input word. Unlike most previous representation learning models in domain adaptation, PBLM can naturally feed structure aware text classifiers such as LSTM and CNN. We experiment with the task of cross-domain sentiment classification on 20 domain pairs and show substantial improvements over strong baselines.1 1 Introduction Domain adaptation (DA, (Daum´e III, 2007; Ben- David et al., 2010)) is a fundamental challenge in NLP, due to the reliance of many algorithms on costly labeled data which is scarce in many do- mains. To save annotation efforts, DA aims to im- port algorithms trained with labeled data from one or several domains to new ones. While DA algo- rithms have long been developed for many tasks and domains (e.g. (Jiang and Zhai, 2007; Mc- Closky et al., 2010; Titov, 2011; Bollegala et al., 2011; Rush et al., 2012; Schnabel and Sch¨utze, 1Our code is publicly available at: https://github. com/yftah89/PBLM-Domain-Adaptation. 2014)), the unprecedented growth of heteroge- neous online content calls for more progress. DA through Representation Learning (DReL), where the DA method induces shared representa- tions for the examples in the source and the tar- get domains, has become prominent in the Neural Network (NN) era. A seminal (non-NN) DReL work is structural correspondence learning (SCL) (Blitzer et al., 2006, 2007) which models the con- nections between pivot features – features that are frequent in the source and the target domains and are highly correlated with the task label in the source domain – and the other, non-pivot, fea- tures. While this approach explicitly models the correspondence between the source and the target domains, it has been outperformed by NN-based models, particularly those based on autoencoders (AEs, (Glorot et al., 2011; Chen et al., 2012)) which employ compress-based noise reduction to extract features that empirically support domain adaptation. Recently, Ziser and Reichart (2017) (ZR17) proposed to marry these approaches. They have presented the autoencoder-SCL models and demonstrated their superiority over a large num- ber of previous approaches, particularly those that employ pivot-based ideas only or NNs only. Current DReL methods, however, suffer from a fundamental limitation: they ignore the struc- ture of their …

Content truncated to 500 words