Revisiting Word Embedding for Contrasting Meaning

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 106–115, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Revisiting Word Embedding for Contrasting Meaning Zhigang Chen†, Wei Lin‡, Qian Chen∗, Xiaoping Chen†, Si Wei‡, Hui Jiang§ and Xiaodan Zhu§ †School of Computer Science and Technology, ∗NELSLIP, University of Science and Technology of China, Hefei, China ‡ iFLYTEK Research, Hefei, China §Department of EECS, York University, Toronto, Canada emails: zgchen9517017@gmail.com, weilin2@iflytek.com, cq1231@mail.ustc.edu.cn, xpchen@ustc.edu.cn, siwei@iflytek.com, hj@cse.yorku.ca, zhu2048@gmail.com Abstract Contrasting meaning is a basic aspect of semantics. Recent word-embedding mod- els based on distributional semantics hy- pothesis are known to be weak for mod- eling lexical contrast. We present in this paper the embedding models that achieve an F-score of 92% on the widely-used, publicly available dataset, the GRE “most contrasting word” questions (Mohammad et al., 2008). This is the highest perfor- mance seen so far on this dataset. Sur- prisingly at the first glance, unlike what was suggested in most previous work, where relatedness statistics learned from corpora is claimed to yield extra gains over lexicon-based models, we obtained our best result relying solely on lexical re- sources (Roget’s and WordNet)—corpora statistics did not lead to further improve- ment. However, this should not be sim- ply taken as that distributional statistics is not useful. We examine several basic con- cerns in modeling contrasting meaning to provide detailed analysis, with the aim to shed some light on the future directions for this basic semantics modeling problem. 1 Introduction Learning good representations of meaning for dif- ferent granularities of texts is core to human lan- guage understanding, where a basic problem is representing the meanings of words. Distributed representations learned with neural networks have recently showed to result in significant improve- ment of performance on a number of language understanding problems (e.g., speech recognition and automatic machine translation) and on many non-language problems (e.g., image recognition). Distributed representations have been leveraged to represent words as in (Collobert et al., 2011; Mikolov et al., 2013). Contrasting meaning is a basic aspect of seman- tics, but it is widely known that word embedding models based on distributional semantics hypoth- esis are weak in modeling this—contrasting mean- ing is often lost in the low-dimensional spaces based on such a hypothesis, and better models would be desirable. Lexical contrast has been modeled in (Lin and Zhao, 2003; Mohammad et al., 2008; Moham- mad et al., 2013). The recent literature has also included research efforts of modeling contrasting meaning in embedding spaces, leading to state- of-the-art performances. For example, Yih et al. (2012) proposed to use polarity-primed latent se- mantic analysis (LSA), called PILSA, to capture contrast, which was further used to initialize a neu- ral network and achieved an F-score of 81% on the same GRE “most contrasting word” questions (Mohammad et al., 2008). More recently, Zhang et al. (2014) proposed a tensor factorization ap- proach to solving the problem, resulting in …

Content truncated to 500 words