Reinforcing the Topic of Embeddings with Theta Pure Dependence for Text Classification

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2551–2556, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Reinforcing the Topic of Embeddings with Theta Pure Dependence for Text Classification∗ Ning Xing1, Yuexian Hou1, Peng Zhang1, Wenjie Li2, Dawei Song1 1School of Computer Science and Technology, Tianjin University, China 2Department of Computing, The Hong Kong Polytechnic University, Hong Kong {xingning,yxhou,pzhang,dwsong}@tju.edu.cn,cswjli@comp.polyu.edu.hk Abstract For sentiment classification, it is often recognized that embedding based on dis- tributional hypothesis is weak in captur- ing sentiment contrast–contrasting words may have similar local context. Based on broader context, we propose to incorporate Theta Pure Dependence (TPD) into the Paragraph Vector method to reinforce top- ical and sentimental information. TPD has a theoretical guarantee that the word de- pendency is pure, i.e., the dependence pat- tern has the integral meaning whose under- lying distribution can not be conditionally factorized. Our method outperforms the state-of-the-art performance on text clas- sification tasks. 1 Introduction Word embeddings can be learned by training a neural probabilistic language model or a uni- fied neural network architecture for various NLP tasks (Bengio et al., 2003; Collobert and We- ston, 2008; Collobert et al., 2011). In global context-aware neural language model (Huang et al., 2012), the global context vector is a weighted average of all word embeddings of a single docu- ment/paragraph. After trained with all word em- beddings belonging to the current paragraph, a re- sulting Paragraph Vector can be obtained. Actu- ally, Le and Mikolov’s Paragraph Vector (Le and Mikolov, 2014) is trained based on the log-linear neural language model (Mikolov et al., 2013a). For text classification, using a straightforward extension of language model (e.g. Le and Mikolov’s Paragraph Vector) is considered not to be sensible. Embeddings learned for text classifi- cation should be very different from that learned for language modeling. For example, language ∗Corresponding authors: Yuexian Hou and Peng Zhang. models often calculate the probability of a sen- tence, therefore this is a good movie and this is a bad movie may not be discriminated from each other. In sentiment analysis task, the semantic representation of words needs to tell word good from bad, even if the two words have the same local context. For this reason, the local depen- dency is insufficient to model topical or sentiment information. Fortunately, if we have the global context of good like interesting or amazing, the sentiment meaning of the embedding will be ex- plicit. However, the training of log-linear neural language model is based on local word dependen- cies (e.g., the co-occurrence of the words in a local window). Thus, Paragraph Vector can not explic- itly model the word dependencies for those words that do not frequently appear in a local window but are actually closely dependent on each other. In this paper, our aim is to extend the Paragraph Vector with global context which can capture topi- cal or sentiment information effectively. However, if one explicitly considers the dependency patterns that are beyond the local …

Content truncated to 500 words