Improving Neural Language Models by Segmenting, Attending, and Predicting the Future

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1483–1493 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1483 Improving Neural Language Models by Segmenting, Attending, and Predicting the Future Hongyin Luo1 Lan Jiang2 Yonatan Belinkov1 James Glass1 1MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA 02139, USA {hyluo, belinkov, glass}@mit.edu 2School of Information Sciences, University of Illinois at Urbana–Champaign Champaign, IL 61820, USA lanj3@illinois.edu Abstract Common language models typically predict the next word given the context. In this work, we propose a method that improves language modeling by learning to align the given con- text and the following phrase. The model does not require any linguistic annotation of phrase segmentation. Instead, we define syn- tactic heights and phrase segmentation rules, enabling the model to automatically induce phrases, recognize their task-specific heads, and generate phrase embeddings in an unsu- pervised learning manner. Our method can easily be applied to language models with dif- ferent network architectures since an indepen- dent module is used for phrase induction and context-phrase alignment, and no change is re- quired in the underlying language modeling network. Experiments have shown that our model outperformed several strong baseline models on different data sets. We achieved a new state-of-the-art performance of 17.4 per- plexity on the Wikitext-103 dataset. Addi- tionally, visualizing the outputs of the phrase induction module showed that our model is able to learn approximate phrase-level struc- tural knowledge without any annotation. 1 Introduction Neural language models are typically trained by predicting the next word given a past context (Ben- gio et al., 2003). However, natural sentences are not constructed as simple linear word sequences, as they usually contain complex syntactic infor- mation. For example, a subsequence of words can constitute a phrase, and two non-neighboring words can depend on each other. These properties make natural sentences more complex than simple linear sequences. Most recent work on neural language modeling learns a model by encoding contexts and match- ing the context embeddings to the embedding of the next word (Bengio et al., 2003; Merity et al., 2017; Melis et al., 2017). In this line of work, a given context is encoded with a neural network, for example a long short-term memory (LSTM; Hochreiter and Schmidhuber, 1997) network, and is represented with a distributed vector. The log- likelihood of predicting a word is computed by calculating the inner product between the word embedding and the context embedding. Although most models do not explicitly consider syntax, they still achieve state-of-the-art performance on different corpora. Efforts have also been made to utilize structural information to learn better language models. For instance, parsing-reading- predict networks (PRPN; Shen et al., 2017) ex- plicitly learn a constituent parsing structure of a sentence and predict the next word considering the internal structure of the given context with an at- tention mechanism. Experiments have shown that the model is able to capture some syntactic infor- mation. Similar to word representation learning models that learns …

Content truncated to 500 words