A Simple and Effective Method for Injecting Word-Level Information into Character-Aware Neural Language Models

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 920–928 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 920 A Simple and Effective Method for Injecting Word-level Information into Character-aware Neural Language Models Yukun Feng1, Hidetaka Kamigaito1, Hiroya Takamura1,2 and Manabu Okumura1 1Tokyo Institute of Technology 2National Institute of Advanced Industrial Science and Technology (AIST) {yukun@lr.,kamigaito@lr.,takamura@,oku@}pi.titech.ac.jp Abstract We propose a simple and effective method to inject word-level information into character- aware neural language models. Unlike pre- vious approaches which usually inject word- level information at the input of a long short- term memory (LSTM) network, we inject it into the softmax function. The resultant model can be seen as a combination of character- aware language model and simple word-level language model. Our injection method can also be used together with previous methods. Through the experiments on 14 typologically diverse languages, we empirically show that our injection method, when used together with the previous methods, works better than the previous methods, including a gating mecha- nism, averaging, and concatenation of word vectors. We also provide a comprehensive comparison of these injection methods. 1 Introduction Language modeling (LM) is an important task in the natural language processing field, with various applications such as speech recognition (Mikolov et al., 2010a), machine translation (Koehn, 2009) and summarization (Filippova et al., 2015). Re- cently, neural language models (NLMs) have shown a great success and are better than tradi- tional count-based methods (Bengio et al., 2003; Mikolov et al., 2010b). Standard NLMs usually maintain a fixed vocabulary and map each word to a continuous representation. These word rep- resentations obtained through NLMs are usually close to each other in the induced vector space if they are semantically similar. However, there are two main problems of standard NLMs. One is that they cannot handle out-of-vocabulary words. These words are usually replaced with a spe- cial unknown symbol. Another problem is that these models are not effective for learning the re- lationships between words for infrequent words. For example, although words “husbandman” and “salesman” share the suffix “man” in their surface forms, standard NLMs cannot capture such infor- mation in obtaining the relationship between the two words. A common way to deal with these is- sues is to use character information of each word to calculate the word representation, and it is of- ten referred to as character-aware NLMs (Ling et al., 2015; Kim et al., 2016; Vania and Lopez, 2017; Gerz et al., 2018). Our research focuses on utilizing advantages of both character-level infor- mation and word-level information in character- aware NLMs. Previous work usually combines word-level in- formation and character-level information at the input of LSTM layers through a gating mecha- nism, or averaging or concatenation of word vec- tors. Because these approaches generally target at the input vectors, the word-level information can- not be explicitly taken into account at the output layer for predicting the next word. To deal with this problem, we propose an im- proved character-aware neural language …

Content truncated to 500 words