Co-learning of Word Representations and Morpheme Representations

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 141–150, Dublin, Ireland, August 23-29 2014. Co-learning of Word Representations and Morpheme Representations Siyu Qiu Nankai University Tianjin, 300071, China ppqq2356@gmail.com Qing Cui Tsinghua University Beijing, 100084, China cuiq12@mails.tsinghua.edu.cn Jiang Bian Microsoft Research Beijing, 100080, China jibian@microsoft.com Bin Gao Microsoft Research Beijing, 100080, China bingao@microsoft.com Tie-Yan Liu Microsoft Research Beijing, 100080, China tyliu@microsoft.com Abstract The techniques of using neural networks to learn distributed word representations (i.e., word embeddings) have been used to solve a variety of natural language processing tasks. The re- cently proposed methods, such as CBOW and Skip-gram, have demonstrated their effectiveness in learning word embeddings based on context information such that the obtained word embed- dings can capture both semantic and syntactic relationships between words. However, it is quite challenging to produce high-quality word representations for rare or unknown words due to their insufficient context information. In this paper, we propose to leverage morphological knowledge to address this problem. Particularly, we introduce the morphological knowledge as both ad- ditional input representation and auxiliary supervision to the neural network framework. As a result, beyond word representations, the proposed neural network model will produce morpheme representations, which can be further employed to infer the representations of rare or unknown words based on their morphological structure. Experiments on an analogical reasoning task and several word similarity tasks have demonstrated the effectiveness of our method in producing high-quality words embeddings compared with the state-of-the-art methods. 1 Introduction Word representation is a key factor for many natural language processing (NLP) applications. In the conventional solutions to the NLP tasks, discrete word representations are often adopted, such as the 1-of-v representations, where v is the size of the entire vocabulary and each word in the vocabulary is represented as a long vector with only one non-zero element. However, using discrete word vectors cannot indicate any relationships between different words, even though they may yield high semantic or syntactic correlations. For example, while careful and carefully have quite similar semantics, their corresponding 1-of-v representations trigger different indexes to be the hot values, and it is not explicit that careful is much closer to carefully than other words using 1-of-v representations. To deal with the problem, neural network models have been widely applied to obtain word repre- sentations. In particular, they usually take the 1-of-v representations as the word input vectors in the neural networks, and learn new distributed word representations in a low-dimensional continuous em- bedding space. The principle of these models is that words that are highly correlated in terms of either semantics or syntactics should be close to each other in the embedding space. Representative works in this field include feed-forward neural network language model (NNLM) (Bengio et al., 2003), recurrent neural network language model (RNNLM) (Mikolov et al., 2010), and the recently proposed continues bag-of-words (CBOW) model and continues skip-gram (Skip-gram) model (Mikolov et al., 2013a). However, there are still challenges for using neural network models to …

Content truncated to 500 words