Multi-prototype Chinese Character Embedding

Multi-prototype Chinese Character Embedding Yanan Lu1, Yue Zhang2, Donghong Ji1 Computer School, Wuhan University, China1 Singapore University of Technology and Design2 {luyanan, dhji}@whu.edu.cn, yuezhang@stud.edu.sg Abstract Chinese sentences are written as sequences of characters, which are elementary units of syntax and semantics. Characters are highly polysemous in forming words. We present a position-sensitive skip-gram model to learn multi-prototype Chinese character embeddings, and explore the usefulness of such character embeddings to Chinese NLP tasks. Evaluation on character similarity shows that multi-prototype embeddings are significantly better than a single-prototype baseline. In addition, used as features in the Chinese NER task, the embeddings result in a 1.74% F-score improvement over a state-of-the-art baseline. Keywords: embedding, multi-prototype, Chinese character 1. Introduction Learned from large-scale unlabeled data, embeddings of- fer distributed vector representations of words and phras- es (Bengio et al., 2003; Mikolov et al., 2013). They have been applied as inputs to neural networks (Collobert and Weston, 2008; Socher et al., 2011; Kalchbrenner et al., 2014), and also used as additional features to improve lin- ear models (Turian et al., 2010; Bansal et al., 2014; Guo et al., 2014b) for NLP tasks. The advantages of embeddings are two-fold. First, they are useful for reducing sparseness compared with discrete words and n-grams. Second, they contain automatically induced features. Written as continuous sequences of characters, Chinese sentences do not have explicit word delimitation. As a re- sult, similar to other Chinese NLP tasks, it is possible to create word embeddings based on automatically segmented Chinese sentences (Zhang et al., 2014b). Although seg- mentation errors can affect the quality, it has been shown that such word embeddings can improve Chinese NLP tasks such as parsing (Wu et al., 2013). On the other hand, character information can also be used for NLP (Zhang et al., 2014a). Zhang et al. (2013) show that parsing can be improved by taking characters as inputs, jointly performing segmentation and syntax parsing. They show that character features can significantly improve the accuracies. Character features are also central to other Chi- nese NLP tasks such as character-based segmentation (X- ue, 2003) and named entity recognition (NER (Chen et al., 2006)). It is an interesting research question whether character em- beddings can be useful for Chinese NLP. However, rela- tively little work has been reported on embedding Chinese characters, and limited effect has been observed (Sun et al., 2014). One possible reason is that the number of characters is much smaller compared to the number of words (104 vs 106), and the effect of character embedding on reducing s- parsity can be limited. Also due to this reason, it has been shown that unsupervised clustering of characters does not improve in-domain segmentation (Liang, 2005). Another possible reason, however, is that previous work did not take into account the polysemy of characters. One u- niqueness of charactes, as compared with words, is that their use is highly flexible, and each character can have multiple meanings in different words. For example, the character “‘” can take at least …

Content truncated to 500 words