Better Word Representations with Recursive Neural Networks for Morphology

Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 104–113, Sofia, Bulgaria, August 8-9 2013. c⃝2013 Association for Computational Linguistics Better Word Representations with Recursive Neural Networks for Morphology Minh-Thang Luong Richard Socher Christopher D. Manning Computer Science Department Stanford University, Stanford, CA, 94305 {lmthang, manning}@stanford.edu richard@socher.org Abstract Vector-space word representations have been very successful in recent years at im- proving performance across a variety of NLP tasks. However, common to most existing work, words are regarded as in- dependent entities without any explicit re- lationship among morphologically related words being modeled. As a result, rare and complex words are often poorly estimated, and all unknown words are represented in a rather crude way using only one or a few vectors. This paper addresses this shortcoming by proposing a novel model that is capable of building representations for morphologically complex words from their morphemes. We combine recursive neural networks (RNNs), where each mor- pheme is a basic unit, with neural language models (NLMs) to consider contextual information in learning morphologically- aware word representations. Our learned models outperform existing word repre- sentations by a good margin on word sim- ilarity tasks across many datasets, includ- ing a new dataset we introduce focused on rare words to complement existing ones in an interesting way. 1 Introduction The use of word representations or word clusters pretrained in an unsupervised fashion from lots of text has become a key “secret sauce” for the suc- cess of many NLP systems in recent years, across tasks including named entity recognition, part-of- speech tagging, parsing, and semantic role label- ing. This is particularly true in deep neural net- work models (Collobert et al., 2011), but it is also true in conventional feature-based models (Koo et al., 2008; Ratinov and Roth, 2009). Deep learning systems give each word a distributed representation, i.e., a dense low- dimensional real-valued vector or an embedding. The main advantage of having such a distributed representation over word classes is that it can cap- ture various dimensions of both semantic and syn- tactic information in a vector where each dimen- sion corresponds to a latent feature of the word. As a result, a distributed representation is compact, less susceptible to data sparsity, and can implicitly represent an exponential number of word clusters. However, despite the widespread use of word clusters and word embeddings, and despite much work on improving the learning of word repre- sentations, from feed-forward networks (Bengio et al., 2003) to hierarchical models (Morin, 2005; Mnih and Hinton, 2009) and recently recurrent neural networks (Mikolov et al., 2010; Mikolov et al., 2011), these approaches treat each full-form word as an independent entity and fail to cap- ture the explicit relationship among morphologi- cal variants of a word.1 The fact that morphologi- cally complex words are often rare exacerbates the problem. Though existing clusterings and embed- dings represent well frequent words, such as “dis- tinct”, they often badly model rare ones, such as “distinctiveness”. In this work, we use recursive neural …

Content truncated to 500 words