Learning Latent Word Representations for Domain Adaptation using Supervised Word Clustering

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 152–162, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Learning Latent Word Representations for Domain Adaptation using Supervised Word Clustering Min Xiao and Feipeng Zhao and Yuhong Guo Department of Computer and Information Sciences Temple University Philadelphia, PA 19122, USA {minxiao,feipeng.zhao,yuhong}@temple.edu Abstract Domain adaptation has been popularly stud- ied on exploiting labeled information from a source domain to learn a prediction model in a target domain. In this paper, we develop a novel representation learning approach to ad- dress domain adaptation for text classification with automatically induced discriminative la- tent features, which are generalizable across domains while informative to the prediction task. Specifically, we propose a hierarchical multinomial Naive Bayes model with latent variables to conduct supervised word cluster- ing on labeled documents from both source and target domains, and then use the produced cluster distribution of each word as its la- tent feature representation for domain adapta- tion. We train this latent graphical model us- ing a simple expectation-maximization (EM) algorithm. We empirically evaluate the pro- posed method with both cross-domain doc- ument categorization tasks on Reuters-21578 dataset and cross-domain sentiment classifica- tion tasks on Amazon product review dataset. The experimental results demonstrate that our proposed approach achieves superior perfor- mance compared with alternative methods. 1 Introduction Supervised prediction models typically require a large amount of labeled data for training. However, manually collecting data annotations is expensive in many real-world applications such as document cat- egorization or sentiment classification. Recently, do- main adaptation has been proposed to exploit exist- ing labeled data in a related source domain to assist the prediction model training in the target domain (Ben-David et al., 2006; Blitzer et al., 2006; Daum´e III, 2007; Blitzer et al., 2011; Chen et al., 2012). As an effective tool to reduce annotation effort, domain adaptation has achieved success in various cross- domain natural language processing (NLP) systems such as document categorization (Dai et al., 2007), sentiment classification (Blitzer et al., 2007; Chen et al., 2012; Mejova and Srinivasan, 2012; Chen et al., 2011), email spam detection (Jiang and Zhai, 2007), and a number of other NLP tasks (Blitzer et al., 2011; Daum´e III, 2007). One primary challenge of domain adaptation lies in the distribution divergence of the two domains in the original feature representation space. For ex- ample, documents about books may contain very different high-frequency words and discriminative words from documents about kitchen. A good cross- domain feature representation thus has been viewed as critical for bridging the domain divergence gap and facilitating domain adaptation in the NLP area (Ben-David et al., 2006, 2010). Many domain adap- tation works have been proposed to learn new cross-domain feature representations (Blitzer et al., 2006, 2011). Though demonstrated good perfor- mance on certain problems, these works mostly in- duce new feature representations in an unsupervised way, without taking the valuable label information into account. In this work, we present a novel supervised rep- resentation …

Content truncated to 500 words