Incorporating Side Information into Recurrent Neural Network Language Models

Proceedings of NAACL-HLT 2016, pages 1250–1255, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Incorporating Side Information into Recurrent Neural Network Language Models Cong Duy Vu Hoang University of Melbourne Melbourne, VIC, Australia vhoang2@student.unimelb.edu.au Gholamreza Haffari Monash University Clayton, VIC, Australia gholamreza.haffari@monash.edu Trevor Cohn University of Melbourne Melbourne, VIC, Australia t.cohn@unimelb.edu.au Abstract Recurrent neural network language models (RNNLM) have recently demonstrated vast potential in modelling long-term dependen- cies for NLP problems, ranging from speech recognition to machine translation. In this work, we propose methods for conditioning RNNLMs on external side information, e.g., metadata such as keywords, description, doc- ument title or topic headline. Our experiments show consistent improvements of RNNLMs using side information over the baselines for two different datasets and genres in two lan- guages. Interestingly, we found that side in- formation in a foreign language can be highly beneficial in modelling texts in another lan- guage, serving as a form of cross-lingual lan- guage modelling. 1 Introduction Neural network approaches to language modelling (LM) have made remarkable performance gains over traditional count-based ngram LMs (Bengio et al., 2003; Mnih and Hinton, 2007; Mikolov et al., 2011). They offer several desirable characteristics, includ- ing the capacity to generalise over large vocabular- ies through the use of vector space representation, and – for recurrent models (Mikolov et al., 2011) – the ability to encode long distance dependencies that are impossible to include with a limited context windows used in conventional ngram LMs. These early papers have spawned a cottage industry in neu- ral LM based applications, where text generation is a key component, including conditional language models for image captioning (Kiros et al., 2014; Vinyals et al., 2015) and neural machine translation (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015). Inspired by these works for conditioning LMs on complex side information, such as images and for- eign text, in this paper we investigate the possibility of improving LMs in a more traditional setting, that is when applied directly to text documents. Typi- cally corpora include rich side information, such as document titles, authorship, time stamp, keywords and so on, although this information is usually dis- carded when applying statistical models. However, this information can be highly informative, for in- stance, keywords, titles or descriptions, often in- clude central topics which will be helpful in mod- elling or understanding the document text. We pro- pose mechanisms for encoding this side informa- tion into a vector space representation, and means of incorporating it into the generating process in a RNNLM framework. Evaluating on two corpora and two different languages, we show consistently sig- nificant perplexity reductions over the state-of-the- art RNNLM models. The contributions of this paper are as follows: 1. We propose a framework for encoding struc- tured and unstructured side information, and its incorporation into a RNNLM. 2. We introduce a new corpus, the RIE corpus, based on the Europarl web archive, with rich annotations of several types of meta-data. 3. We provide …

Content truncated to 500 words