Improve Chinese Word Embeddings by Exploiting Internal Structure

Proceedings of NAACL-HLT 2016, pages 1041–1050, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Improve Chinese Word Embeddings by Exploiting Internal Structure Jian Xu, Jiawei Liu, Liangang Zhang, Zhengyu Li, Huanhuan Chen∗ Department of Computer Science, University of Science and Technology of China, China {jianxu1, ustcljw, liangang, lzy0503}@mail.ustc.edu.cn hchen@ustc.edu.cn Abstract Recently, researchers have demonstrated that both Chinese word and its component charac- ters provide rich semantic information when learning Chinese word embeddings. Howev- er, they ignored the semantic similarity across component characters in a word. In this paper, we learn the semantic contribution of charac- ters to a word by exploiting the similarity be- tween a word and its component characters with the semantic knowledge obtained from other languages. We propose a similarity- based method to learn Chinese word and char- acter embeddings jointly. This method is also capable of disambiguating Chinese character- s and distinguishing non-compositional Chi- nese words. Experiments on word similarity and text classification demonstrate the effec- tiveness of our method. 1 Introduction Distributed representations of knowledge has re- ceived wide attention in recent years. Researchers have proposed various models to learn it at different granularity levels. Distributed word representation- s, also known as word embeddings, were learned in (Rumelhart et al., 1988; Bengio et al., 2006; Mni- h and Hinton, 2009; Mikolov et al., 2013a). Larger granularity levels than words have also been inves- tigated, including phrase level (Socher et al., 2010; Zhang et al., 2014; Yu and Dredze, 2015), sentence level (Le and Mikolov, 2014; Socher et al., 2013; Kalchbrenner et al., 2014; Kiros et al., 2015), and ∗Corresponding author document level (Le and Mikolov, 2014; Hermann and Blunsom, 2014; Srivastava et al., 2013). For language like Chinese, some smaller units than word also provide rich semantic information. For example, Chinese characters in word, Chinese radicals in character. These internal structures have been proved to be useful for Chinese word and char- acter embeddings (Chen et al., 2015; Li et al., 2015). Chen et al. (2015) took Chinese characters in a word into account when modeling the semantic meaning of the word. They proposed a character-enhanced word embeddings model (CWE) by adding the em- bedding of component characters in a word with the same weight to the word embedding. However, the internal characters in a Chinese word have dif- ferent semantic contributions to its meaning. Take Chinese word ““�” (frog) as an example. The character ““” (blue or green) is to decorate char- acter “�” (frog). It is obvious that the latter char- acter contributes more than the former one to the word meaning. In Li et al. (2015), they proposed a component-enhanced Chinese character embed- dings model based on the feature that most Chinese characters are phono-semantic compounds. They considered characters and bi-characters as the ba- sic embedding units. However, some bi-characters are meaningless, and may not form a Chinese word. These bi-characters may undermine embeddings of others. This paper, motivated by Chen et al. (2015), ex- ploits the internal structures …

Content truncated to 500 words