Learning to Generate Word Representations using Subword Information

Proceedings of the 27th International Conference on Computational Linguistics, pages 2551–2561 Santa Fe, New Mexico, USA, August 20-26, 2018. 2551 Learning to Generate Word Representations using Subword Information Yeachan Kim, Kang-Min Kim, Ji-Min Lee and SangKeun Lee Korea University, Seoul, Republic of Korea {yeachan,kangmin89,jm94318,yalphy}@korea.ac.kr Abstract Distributed representations of words play a major role in the field of natural language process- ing by encoding semantic and syntactic information of words. However, most existing works on learning word representations typically regard words as individual atomic units and thus are blind to subword information in words. This further gives rise to a difficulty in representing out-of-vocabulary (OOV) words. In this paper, we present a character-based word representation approach to deal with these limitations. The proposed model learns to generate word representa- tions from characters. In our model, we employ a convolutional neural network and a highway network over characters to extract salient features effectively. Unlike previous models that learn word representations from a large corpus, we take a set of pre-trained word embeddings and generalize it to word entries, including OOV words. To demonstrate the efficacy of the proposed model, we perform both an intrinsic and an extrinsic task which are word similarity and language modeling, respectively. Experimental results show clearly that the proposed model significantly outperforms strong baseline models that regard words or their subwords as atomic units. For ex- ample, we achieve as much as 18.5% improvement on average in perplexity for morphologically rich languages compared to strong baselines in the language modeling task. 1 Introduction Recently, distributed representations of words have become an inevitable component in systems of nat- ural language processing (NLP), e.g., chunking and named entity recognition (Turian et al., 2010), text classification (Kim, 2014), and question answering (Kumar et al., 2016). In such systems, word represen- tations play a vital role by capturing useful information of words, leading to the improved performance of the systems. Most existing approaches to learning word representations are learned in an unsuper- vised learning manner from a large corpus and, for simplicity and effectiveness, they regard words as individual atomic units in a learning phase (Bengio et al., 2003; Collobert and Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014). However, word-based approaches typically ignore subword information in words. It prevents the ap- proaches from taking into account the characteristic of words, “words are lexically related when they share some morphemes or semantic elements” (Williams, 1981). This constraint further makes repre- senting out-of-vocabulary (OOV) words difficult. Even if lexically related words have been observed, word-based approaches fail to represent such words that are not included in a pre-defined vocabulary. Perhaps the simplest way to sidestep this problem is to use one or more pseudo-words (e.g., <unk>) to represent OOV words. However, this is not a fundamental remedy, as OOV words have their own meanings in many cases, such as variants, compounds, and misspellings. This problem is more evident when the domain of a text in which the word is learned differs …

Content truncated to 500 words