Attentive Language Models

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 441–450, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Attentive Language Models Giancarlo D. Salton and Robert J. Ross and John D. Kelleher Applied Intelligence Research Centre School of Computing Dublin Institute of Technology Ireland giancarlo.salton@mydit.ie {robert.ross,john.d.kelleher}@dit.ie Abstract In this paper, we extend Recurrent Neural Network Language Models (RNN-LMs) with an attention mechanism. We show that an Attentive RNN-LM (with 14.5M parameters) achieves a better perplexity than larger RNN-LMs (with 66M param- eters) and achieves performance compa- rable to an ensemble of 10 similar sized RNN-LMs. We also show that an Attentive RNN-LM needs less contextual informa- tion to achieve similar results to the state- of-the-art on the wikitext2 dataset. 1 Introduction Language Models (LMs) are an essential compo- nent in a range of Natural Language Processing applications, such as Statistical Machine Trans- lation and Speech Recognition (Schwenk et al., 2012). An LM provides a probability for a se- quence of words in a given language, reflecting fluency and the likelihood of that word sequence occurring in that language. In recent years Recurrent Neural Networks (RNNs) have improved the state-of-the-art in LM research (J´ozefowicz et al., 2016). Sequential data prediction, however, is still considered a challenge in Artificial Intelligence (Mikolov et al., 2010) given that, in general, prediction accuracy de- grades as the size of sequences increase. RNN-LMs sequentially propagate forward a context vector by integrating the information gen- erated by each prediction step into the context used for the next prediction. One consequence of this forward propagation of information is that older information tends to fade from the context as new information is integrated into the context. As a re- sult, RNN-LMs struggle in situations where there is a long-distance dependency because the relevant information from the start of the dependency has faded by the time the model has spanned the de- pendency. A second problem is that the context can be dominated by the more recent information so when an RNN-LM does make an error this error can be propagated forward resulting in a cascade of errors through the rest of the sequence. In recent sequence-to-sequence research the concept of “attention” has been developed to en- able RNNs to align different parts of the input and output sequences. Examples of attention based architectures include Neural Machine Translation (NMT) (Bahdanau et al., 2015; Luong et al., 2015) and image captioning (Xu et al., 2015). In this paper we extend the RNN-LM context mechanism with an attention mechanism that en- ables the model to bring forward context infor- mation from different points in the context se- quence history. We hypothesis that this atten- tion mechanism enables RNN-LMs to: (a) bridge long-distance dependencies, thereby avoiding er- rors; and, (b) to overlook recent errors by choos- ing to focus on contextual information preceding the error, thereby avoiding error propagation. We show that a medium sized1 Attentive RNN- LM2 achieves better performance than larger “standard” models and performance …

Content truncated to 500 words