Learning Semantic Word Embeddings based on Ordinal Knowledge Constraints

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1501–1511, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Learning Semantic Word Embeddings based on Ordinal Knowledge Constraints Quan Liu† and Hui Jiang‡ and Si Wei§ and Zhen-Hua Ling† and Yu Hu§ † National Engineering Laboratory for Speech and Language Information Processing University of Science and Technology of China, Hefei, China ‡ Department of Electrical Engineering and Computer Science, York University, Canada § iFLYTEK Research, Hefei, China emails: quanliu@mail.ustc.edu.cn, hj@cse.yorku.ca, siwei@iflytek.com, zhling@ustc.edu.cn, yuhu@iflytek.com Abstract In this paper, we propose a general frame- work to incorporate semantic knowledge into the popular data-driven learning pro- cess of word embeddings to improve the quality of them. Under this framework, we represent semantic knowledge as many ordinal ranking inequalities and formu- late the learning of semantic word embed- dings (SWE) as a constrained optimiza- tion problem, where the data-derived ob- jective function is optimized subject to all ordinal knowledge inequality constraints extracted from available knowledge re- sources such as Thesaurus and Word- Net. We have demonstrated that this con- strained optimization problem can be ef- ficiently solved by the stochastic gradient descent (SGD) algorithm, even for a large number of inequality constraints. Experi- mental results on four standard NLP tasks, including word similarity measure, sen- tence completion, name entity recogni- tion, and the TOEFL synonym selection, have all demonstrated that the quality of learned word vectors can be significantly improved after semantic knowledge is in- corporated as inequality constraints during the learning process of word embeddings. 1 Introduction Distributed word representation (i.e., word embed- ding) is a technique that represents words as con- tinuous vectors, which is an important research topic in natural language processing (NLP) (Hin- ton et al., 1986; Turney et al., 2010). In re- cent years, it has been widely used in various NLP tasks, including neural language model (Ben- gio et al., 2003; Schwenk, 2007), sequence la- belling tasks (Collobert and Weston, 2008; Col- lobert et al., 2011), machine translation (Devlin et al., 2014; Sutskever et al., 2014), and antonym se- lection (Chen et al., 2015). Typically, word vectors are learned based on the distributional hypothesis (Harris, 1954; Miller and Charles, 1991), which assumes that words with a similar context tend to have a similar meaning. Under this hypothe- sis, various models, such as the skip-gram model (Mikolov et al., 2013a; Mikolov et al., 2013b; Levy and Goldberg, 2014) and GloVe model (Pen- nington et al., 2014), have been proposed to lever- age the context of each word in large corpora to learn word embeddings. These methods can ef- ficiently estimate the co-occurrence statistics to model contextual distributions from very large text corpora and they have been demonstrated to be quite effective in a number of NLP tasks. How- ever, they still suffer from some major limitations. In particular, these corpus-based methods usu- ally fail to capture the precise meanings for many words. For example, …

Content truncated to 500 words