Joint Embeddings of Chinese Words, Characters, and Fine-grained Subcharacter Components

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 286–291 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Joint Embeddings of Chinese Words, Characters, and Fine-grained Subcharacter Components Jinxing Yu Xun Jian Hao Xin Yangqiu Song Department of Computer Science and Engineering The Hong Kong University of Science and Technology {jyuat, xjian, hxinaa, yqsong}@cse.ust.hk Abstract Word embeddings have attracted much at- tention recently. Different from alpha- betic writing systems, Chinese characters are often composed of subcharacter com- ponents which are also semantically infor- mative. In this work, we propose an ap- proach to jointly embed Chinese words as well as their characters and fine-grained subcharacter components. We use three likelihoods to evaluate whether the con- text words, characters, and components can predict the current target word, and collected 13,253 subcharacter components to demonstrate the existing approaches of decomposing Chinese characters are not enough. Evaluation on both word similar- ity and word analogy tasks demonstrates the superior performance of our model. 1 Introduction Distributed word representation represents a word as a vector in a continuous vector space and can better uncover both the semantic and syntactic in- formation over traditional one-hot representations. It has been successfully applied to many down- stream natural language processing (NLP) tasks as input features, such as named entity recog- nition (Collobert et al., 2011), text classification (Joulin et al., 2016), sentiment analysis (Tang et al., 2014), and question answering (Zhou et al., 2015). Among many embedding methods (Ben- gio et al., 2003; Mnih and Hinton, 2009), CBOW and Skip-Gram models are very popular due to their simplicity and efficiency, making it feasi- ble to learn good embeddings of words from large scale training corpora (Mikolov et al., 2013b,a). Despite the success and popularity of word em- beddings, most of the existing methods treat each word as the minimum unit, which ignores the mor- phological information of words. Rare words can- not be well represented when optimizing a cost function related to a rare word and its contexts. To address this issue, some recent studies (Luong et al., 2013; Qiu et al., 2014; Sun et al., 2016a; Wieting et al., 2016) have investigated how to ex- ploit morphemes or character n-grams to learn bet- ter embeddings of English words. Different from other alphabetic writing systems such as English, written Chinese is logosyllabic, i.e., a Chinese character can be a word on its own or part of a polysyllabic word1. The characters them- selves are often composed of subcharacter com- ponents which are also semantically informative. The subword items of Chinese words, including characters and subcharacter components, contain rich semantic information. The characters com- posing a word can indicate the semantic mean- ing of the word and the subcharacter components, such as radicals and components themselves being a character, composing a character can indicate the semantic meaning of the character. The compo- nents of characters can be roughly divided into two types: semantic component and phonetic compo- nent. The semantic component indicates …

Content truncated to 500 words