Unsupervised Cross-Domain Word Representation Learning

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 730–740, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Unsupervised Cross-Domain Word Representation Learning Danushka Bollegala Takanori Maehara Ken-ichi Kawarabayashi danushka.bollegala@ maehara.takanori@ k keniti@ liverpool.ac.uk shizuoka.ac.jp nii.ac.jp University of Liverpool Shizuoka University National Institute of Informatics JST, ERATO, Kawarabayashi Large Graph Project. Abstract Meaning of a word varies from one do- main to another. Despite this impor- tant domain dependence in word seman- tics, existing word representation learning methods are bound to a single domain. Given a pair of source-target domains, we propose an unsupervised method for learning domain-specific word representa- tions that accurately capture the domain- specific aspects of word semantics. First, we select a subset of frequent words that occur in both domains as pivots. Next, we optimize an objective function that enforces two constraints: (a) for both source and target domain documents, piv- ots that appear in a document must accu- rately predict the co-occurring non-pivots, and (b) word representations learnt for pivots must be similar in the two do- mains. Moreover, we propose a method to perform domain adaptation using the learnt word representations. Our proposed method significantly outperforms compet- itive baselines including the state-of-the- art domain-insensitive word representa- tions, and reports best sentiment classifi- cation accuracies for all domain-pairs in a benchmark dataset. 1 Introduction Learning semantic representations for words is a fundamental task in NLP that is required in nu- merous higher-level NLP applications (Collobert et al., 2011). Distributed word representations have gained much popularity lately because of their accuracy as semantic representations for words (Mikolov et al., 2013a; Pennington et al., 2014). However, the meaning of a word often varies from one domain to another. For exam- ple, the phrase lightweight is often used in a posi- tive sentiment in the portable electronics domain because a lightweight device is easier to carry around, which is a positive attribute for a portable electronic device. However, the same phrase has a negative sentiment assocition in the movie domain because movies that do not invoke deep thoughts in viewers are considered to be lightweight (Bol- legala et al., 2014). However, existing word rep- resentation learning methods are agnostic to such domain-specific semantic variations of words, and capture semantics of words only within a single domain. To overcome this problem and capture domain-specific semantic orientations of words, we propose a method that learns separate dis- tributed representations for each domain in which a word occurs. Despite the successful applications of dis- tributed word representation learning meth- ods (Pennington et al., 2014; Collobert et al., 2011; Mikolov et al., 2013a) most existing ap- proaches are limited to learning only a single representation for a given word (Reisinger and Mooney, 2010). Although there have been some work on learning multiple prototype representa- tions (Huang et al., 2012; Neelakantan et al., 2014) for a word considering its multiple senses, such methods do not consider the …

Content truncated to 500 words