Improved Word Representation Learning with Sememes

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2049–2058 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1187 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2049–2058 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1187 Improved Word Representation Learning with Sememes Yilin Niu1∗, Ruobing Xie1∗, Zhiyuan Liu1,2 †, Maosong Sun1,2 1 Department of Computer Science and Technology, State Key Lab on Intelligent Technology and Systems, National Lab for Information Science and Technology, Tsinghua University, Beijing, China 2 Jiangsu Collaborative Innovation Center for Language Ability, Jiangsu Normal University, Xuzhou 221009 China Abstract Sememes are minimum semantic units of word meanings, and the meaning of each word sense is typically composed by sev- eral sememes. Since sememes are not ex- plicit for each word, people manually an- notate word sememes and form linguistic common-sense knowledge bases. In this paper, we present that, word sememe in- formation can improve word representa- tion learning (WRL), which maps word- s into a low-dimensional semantic space and serves as a fundamental step for many NLP tasks. The key idea is to utilize word sememes to capture exact meanings of a word within specific contexts accu- rately. More specifically, we follow the framework of Skip-gram and present three sememe-encoded models to learn repre- sentations of sememes, senses and word- s, where we apply the attention scheme to detect word senses in various contexts. We conduct experiments on two tasks in- cluding word similarity and word analogy, and our models significantly outperform baselines. The results indicate that WRL can benefit from sememes via the attention scheme, and also confirm our models be- ing capable of correctly modeling sememe information. 1 Introduction Sememes are defined as minimum semantic u- nits of word meanings, and there exists a lim- ited close set of sememes to compose the se- mantic meanings of an open set of concepts (i.e. word sense). However, sememes are not explicit ∗indicates equal contribution †Corresponding author: Z. Liu (liuzy@tsinghua.edu.cn) for each word. Hence, people manually annotate word sememes and build linguistic common-sense knowledge bases. HowNet (Dong and Dong, 2003) is one of such knowledge bases, which annotates each concep- t in Chinese with one or more relevant sememes. Different from WordNet (Miller, 1995), the phi- losophy of HowNet emphasizes the significance of part and attribute represented by sememes. HowNet has been widely utilized in word similar- ity computation (Liu and Li, 2002) and sentiment analysis (Xianghua et al., 2013), and in section 3.2 we will give a detailed introduction to sememes, senses and words in HowNet. In this paper, we aim to incorporate word se- memes into word representation learning (WRL) and learn improved word embeddings in a low- dimensional semantic space. WRL is a fundamen- tal and critical step in many NLP tasks such as lan- guage modeling (Bengio et al., 2003) and neural machine translation (Sutskever et al., 2014). There have been a lot …

Content truncated to 500 words