Gated Word-Character Recurrent Language Model

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1992–1997, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Gated Word-Character Recurrent Language Model Yasumasa Miyamoto Center for Data Science New York University yasumasa.miyamoto@nyu.edu Kyunghyun Cho Courant Institute of Mathematical Sciences & Centre for Data Science New York University kyunghyun.cho@nyu.edu Abstract We introduce a recurrent neural network lan- guage model (RNN-LM) with long short- term memory (LSTM) units that utilizes both character-level and word-level inputs. Our model has a gate that adaptively finds the op- timal mixture of the character-level and word- level inputs. The gate creates the final vec- tor representation of a word by combining two distinct representations of the word. The character-level inputs are converted into vec- tor representations of words using a bidirec- tional LSTM. The word-level inputs are pro- jected into another high-dimensional space by a word lookup table. The final vector rep- resentations of words are used in the LSTM language model which predicts the next word given all the preceding words. Our model with the gating mechanism effectively utilizes the character-level inputs for rare and out-of- vocabulary words and outperforms word-level language models on several English corpora. 1 Introduction Recurrent neural networks (RNNs) achieve state-of- the-art performance on fundamental tasks of natural language processing (NLP) such as language model- ing (RNN-LM) (Józefowicz et al., 2016; Zoph et al., 2016). RNN-LMs are usually based on the word- level information or subword-level information such as characters (Mikolov et al., 2012), and predictions are made at either word level or subword level re- spectively. In word-level LMs, the probability distribution over the vocabulary conditioned on preceding words is computed at the output layer using a softmax func- tion. 1 Word-level LMs require a predefined vocab- ulary size since the computational complexity of a softmax function grows with respect to the vocab- ulary size. This closed vocabulary approach tends to ignore rare words and typos, as the words do not appear in the vocabulary are replaced with an out- of-vocabulary (OOV) token. The words appearing in vocabulary are indexed and associated with high- dimensional vectors. This process is done through a word lookup table. Although this approach brings a high degree of freedom in learning expressions of words, infor- mation about morphemes such as prefix, root, and suffix is lost when the word is converted into an index. Also, word-level language models require some heuristics to differentiate between the OOV words, otherwise it assigns the exactly same vector to all the OOV words. These are the major limita- tions of word-level LMs. In order to alleviate these issues, we introduce an RNN-LM that utilizes both character-level and word-level inputs. In particular, our model has a gate that adaptively choose between two distinct ways to represent each word: a word vector derived from the character-level information and a word vector stored in the word lookup table. This gate is trained to make this decision based on the input word. According to the experiments, …

Content truncated to 500 words