Larger-Context Language Modelling with Recurrent Neural Network

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1319–1329, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Larger-Context Language Modelling with Recurrent Neural Network∗ Tian Wang Center for Data Science New York University t.wang@nyu.edu Kyunghyun Cho Courant Institute of Mathematical Sciences and Center for Data Science New York University kyunghyun.cho@nyu.edu Abstract In this work, we propose a novel method to incorporate corpus-level discourse infor- mation into language modelling. We call this larger-context language model. We in- troduce a late fusion approach to a recur- rent language model based on long short- term memory units (LSTM), which helps the LSTM unit keep intra-sentence depen- dencies and inter-sentence dependencies separate from each other. Through the evaluation on four corpora (IMDB, BBC, Penn TreeBank, and Fil9), we demonstrate that the proposed model improves per- plexity significantly. In the experiments, we evaluate the proposed approach while varying the number of context sentences and observe that the proposed late fusion is superior to the usual way of incorporat- ing additional inputs to the LSTM. By an- alyzing the trained larger-context language model, we discover that content words, in- cluding nouns, adjectives and verbs, bene- fit most from an increasing number of con- text sentences. This analysis suggests that larger-context language model improves the unconditional language model by cap- turing the theme of a document better and more easily. 1 Introduction The goal of language modelling is to estimate the probability distribution of various linguistic units, e.g., words, sentences (Rosenfeld, 2000). Among the earliest techniques were count-based n-gram language models which intend to assign the prob- ability distribution of a given word observed af- ∗Recently, (Ji et al., 2015) independently proposed a sim- ilar approach. ter a fixed number of previous words. Later Ben- gio et al. (2003) proposed feed-forward neural language model, which achieved substantial im- provements in perplexity over count-based lan- guage models. Bengio et al. showed that this neu- ral language model could simultaneously learn the conditional probability of the latest word in a se- quence as well as a vector representation for each word in a predefined vocabulary. Recently recurrent neural networks have be- come one of the most widely used models in lan- guage modelling (Mikolov et al., 2010). Long short-term memory unit (LSTM, Hochreiter and Schmidhuber, 1997) is one of the most common recurrent activation function. Architecturally, the memory state and output state are explicitly sep- arated by activation gates such that the vanish- ing gradient and exploding gradient problems de- scribed in Bengio et al. (1994) is avoided. Moti- vated by such gated model, a number of variants of RNNs (e.g. Cho et al. (GRU, 2014b), Chung et al. (GF-RNN, 2015)) have been designed to eas- ily capture long-term dependencies. When modelling a corpus, these language mod- els assume the mutual independence among sen- tences, and the task is often reduced to as- signing a probability to a single sentence. In this work, we propose a method to incorporate corpus-level discourse dependency …

Content truncated to 500 words