Online Representation Learning in Recurrent Neural Language Models

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 238–243, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Online Representation Learning in Recurrent Neural Language Models Marek Rei The ALTA Institute Computer Laboratory University of Cambridge United Kingdom marek.rei@cl.cam.ac.uk Abstract We investigate an extension of continuous online learning in recurrent neural network language models. The model keeps a sep- arate vector representation of the current unit of text being processed and adaptively adjusts it after each prediction. The initial experiments give promising results, indi- cating that the method is able to increase language modelling accuracy, while also decreasing the parameters needed to store the model along with the computation re- quired at each step. 1 Introduction In recent years, neural network models have shown impressive performance on many natural language processing tasks, such as speech recogni- tion (Chorowski et al., 2014; Graves et al., 2013), machine translation (Kalchbrenner and Blunsom, 2013; Cho et al., 2014), text classification (Le and Mikolov, 2014; Kalchbrenner et al., 2014) and image description generation (Kiros et al., 2014). One of the main advantages of these methods is the ability to learn smooth vector representations for words, thereby reducing the sparsity problem inherent in any natural language dataset. Language modelling is another task where neu- ral networks have delivered excellent results (Ben- gio et al., 2003; Mikolov et al., 2011). Chelba et al. (2014) have recently benchmarked several well-known language models by training on very large datasets. They found that a recurrent neu- ral network language model (RNNLM) combined with a 9-gram MaxEnt model was able to give the best results and lowest perplexity. In this work we investigate a possible extension of RNNLM, by allowing it to continue learning and adapting during testing. The model keeps a vector representation of the current sentence that is being processed, and continuously modifies it based on an error signal. We refer to this as a ver- sion of online learning, as gradient descent is used to optimise the vector even during testing. The technique is inspired by work on represen- tation learning (Collobert and Weston, 2008; Mnih and Hinton, 2008; Mikolov et al., 2013), espe- cially Le and Mikolov (2014) who use a related model to learn representations for text classifica- tion. We extend the idea to recurrent models and apply it to the task of language modelling. Our results indicate that by exchanging some existing model parameters for a component using online learning, the system is able to achieve lower per- plexity while also reducing the necessary compu- tation. 2 RNNLM We base our implementation of the RNNLM on Mikolov et al. (2011), shown in Figure 1. The in- put layer to the network consists of a 1-hot vec- tor representing the previous word in the sequence, and the hidden vector from the previous time step. These are multiplied by corresponding weight ma- trices and the resulting vectors are passed through an activation function to calculate the hidden vec- Figure 1: …

Content truncated to 500 words