VCWE: Visual Character-Enhanced Word Embeddings

Proceedings of NAACL-HLT 2019, pages 2710–2719 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2710 VCWE: Visual Character-Enhanced Word Embeddings Chi Sun, Xipeng Qiu∗, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {sunc17,xpqiu,xjhuang}@fudan.edu.cn Abstract Chinese is a logographic writing system, and the shape of Chinese characters contain rich syntactic and semantic information. In this pa- per, we propose a model to learn Chinese word embeddings via three-level composition: (1) a convolutional neural network to extract the intra-character compositionality from the vi- sual shape of a character; (2) a recurrent neural network with self-attention to compose char- acter representation into word embeddings; (3) the Skip-Gram framework to capture non- compositionality directly from the contextual information. Evaluations demonstrate the su- perior performance of our model on four tasks: word similarity, sentiment analysis, named en- tity recognition and part-of-speech tagging.1 1 Introduction Distributed representations of words, namely word embeddings, encode both semantic and syntac- tic information into a dense vector. Currently, word embeddings have been playing a pivotal role in many natural language processing (NLP) tasks. Most of these NLP tasks also benefit from the pre-trained word embeddings, such as word2vec (Mikolov et al., 2013a) and GloVe (Pen- nington et al., 2014), which are based on the dis- tributional hypothesis (Harris, 1954): words that occur in the same contexts tend to have similar meanings. Earlier word embeddings often take a word as a basic unit, and they ignore composi- tionality of its sub-word information such as mor- phemes and character n-grams, and cannot compe- tently handle the rare words. To improve the per- formance of word embeddings, sub-word informa- tion has been employed (Luong et al., 2013; Qiu ∗Corresponding author. 1The source codes are available at https://github. com/HSLCY/VCWE et al., 2014; Cao and Rei, 2016; Sun et al., 2016a; Wieting et al., 2016; Bojanowski et al., 2016). Compositionality is more critical for Chinese, since Chinese is a logographic writing system. In Chinese, each word typically consists of fewer characters and each character also contains richer semantic information. For example, Chinese char- acter “休” (rest) is composed of the characters for “人” (person) and “木” (tree), with the intended idea of someone leaning against a tree, i.e., rest- ing. Based on the linguistic features of Chinese, re- cent methods have used the character informa- tion to improve Chinese word embeddings. These methods can be categorized into two kinds: 1) One kind of methods learn word embeddings with its constituent character (Chen et al., 2015), radical2 (Shi et al., 2015; Yin et al., 2016; Yu et al., 2017) or strokes3 (Cao et al., 2018). How- ever, these methods usually use simple operations, such as averaging and n-gram, to model the inher- ent compositionality within a word, which is not enough to handle the complicated linguistic com- positionality. 2) The other kind of methods learns word em- beddings with the visual information of the char- acter. Liu …

Content truncated to 500 words