Learning Word Representations with Regularization from Prior Knowledge

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 143–152, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Learning Word Representations with Regularization from Prior Knowledge Yan Song Tencent AI Lab clksong@tencent.com Chia-Jung Lee Microsoft cjlee@microsoft.com Fei Xia University of Washington fxia@uw.edu Abstract Conventional word embeddings are train- ed with specific criteria (e.g., based on language modeling or co-occurrence) in- side a single information source, disre- garding the opportunity for further calibra- tion using external knowledge. This paper presents a unified framework that lever- ages pre-learned or external priors, in the form of a regularizer, for enhancing con- ventional language model-based embed- ding learning. We consider two types of regularizers. The first type is derived from topic distribution by running latent Dirich- let allocation on unlabeled data. The sec- ond type is based on dictionaries that are created with human annotation efforts. To effectively learn with the regularizers, we propose a novel data structure, trajectory softmax, in this paper. The resulting em- beddings are evaluated by word similarity and sentiment classification. Experimental results show that our learning framework with regularization from prior knowledge improves embedding quality across multi- ple datasets, compared to a diverse collec- tion of baseline methods. 1 Introduction Distributed representation of words (or word em- bedding) has been demonstrated to be effec- tive in many natural language processing (NLP) tasks (Bengio et al., 2003; Collobert and We- ston, 2008; Turney and Pantel, 2010; Collobert et al., 2011; Mikolov et al., 2013b,d; Weston et al., 2015). Conventional word embeddings are trained with a single objective function (e.g., lan- guage modeling (Mikolov et al., 2013c) or word co-occurrence factorization (Pennington et al., 2014)), which restricts the capability of the learned embeddings from integrating other types of knowledge. Prior work has leveraged relevant sources to obtain embeddings that are best suited for the target tasks, such as Maas et al. (2011) us- ing a sentiment lexicon to enhance embeddings for sentiment classification. However, learning word embeddings with a particular target makes the ap- proach less generic, also implying that customized adaptation has to be made whenever a new knowl- edge source is considered. Along the lines of improving embedding qual- ity, semantic resources have been incorporated as guiding knowledge to refine objective functions in a joint learning framework (Bian et al., 2014; Xu et al., 2014; Yu and Dredze, 2014; Nguyen et al., 2016), or used for retrofitting based on word re- lations defined in the semantic lexicons (Faruqui et al., 2015; Kiela et al., 2015). These approaches, nonetheless, require explicit word relations de- fined in semantic resources, which is a difficult prerequisite for knowledge preparation. Given the above challenges, we propose a novel framework that extends typical context learning by integrating external knowledge sources for en- hancing embedding learning. Compared to a well known work by Faruqui et al. (2015) that focused on tackling the task using a retrofitting1 frame- work on semantic lexicons, our method has an emphasis on …

Content truncated to 500 words