Low-Rank RNN Adaptation for Context-Aware Language Modeling

Low-Rank RNN Adaptation for Context-Aware Language Modeling Aaron Jaech and Mari Ostendorf Department of Electrical Engineering, University of Washington 185 Stevens Way, Paul Allen Center AE100R, Seattle, WA {ajaech,ostendor}@uw.edu Abstract A context-aware language model uses loca- tion, user and/or domain metadata (context) to adapt its predictions. In neural language models, context information is typically rep- resented as an embedding and it is given to the RNN as an additional input, which has been shown to be useful in many applications. We introduce a more powerful mechanism for us- ing context to adapt an RNN by letting the context vector control a low-rank transforma- tion of the recurrent layer weight matrix. Ex- periments show that allowing a greater frac- tion of the model parameters to be adjusted has benefits in terms of perplexity and classi- fication for several different types of context. 1 Introduction In many language modeling applications, the speech or text is associated with some metadata or contex- tual information. For example, in speech recogni- tion, if a user is speaking to a personal assistant then the system might know the time of day or the iden- tity of the task that the user is trying to accomplish. If the user takes a picture of a sign to translate it with their smart phone, the system would have contextual information related to the geographic location and the user’s preferred language. The context-aware language model targets these types of applications with a model that can adapt its predictions based on the provided contextual information. There has been much work on using context infor- mation to adapt language models. Here, we are in- terested in contexts described by metadata (vs. word history or related documents) and in neural network approaches due to their flexibility for representing diverse types of contexts. Specifically, we focus on recurrent neural networks (RNNs) due to their widespread use. The standard approach to adapt an RNN language model is to concatenate the context representation with the word embedding at the input to the RNN (Mikolov and Zweig, 2012). Optionally, the con- text embedding is also concatenated with the out- put from the recurrent layer to adapt the softmax layer. This basic strategy has been adopted for var- ious types of adaptation such as for LM personal- ization (Wen et al., 2013; Li et al., 2016), adapting to television show genres (Chen et al., 2015), and adapting to long range dependencies in a document (Ji et al., 2016), etc. We propose a more powerful mechanism for us- ing a context vector, which we call the FactorCell. Rather than simply using context as an additional input, it is used to control a factored (low-rank) transformation of the recurrent layer weight matrix. The motivation is that allowing a greater fraction of the model parameters to be adjusted in response to the input context will produce a model that is more adaptable and responsive to that context. We evaluate the resulting models in terms of context-dependent perplexity and context classifica- tion …

Content truncated to 500 words