A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models

Proceedings of NAACL-HLT 2016, pages 332–342, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics A Latent Variable Recurrent Neural Network for Discourse Relation Language Models Yangfeng Ji Georgia Institute of Technology Atlanta, GA 30308, USA jiyfeng@gatech.edu Gholamreza Haffari Monash University Clayton, VIC, Australia gholamreza.haffari @monash.edu Jacob Eisenstein Georgia Institute of Technology Atlanta, GA 30308, USA jacobe@gatech.edu Abstract This paper presents a novel latent variable re- current neural network architecture for jointly modeling sequences of words and (possibly latent) discourse relations between adjacent sentences. A recurrent neural network gen- erates individual words, thus reaping the ben- efits of discriminatively-trained vector repre- sentations. The discourse relations are rep- resented with a latent variable, which can be predicted or marginalized, depending on the task. The resulting model can therefore em- ploy a training objective that includes not only discourse relation classification, but also word prediction. As a result, it outperforms state-of- the-art alternatives for two tasks: implicit dis- course relation classification in the Penn Dis- course Treebank, and dialog act classification in the Switchboard corpus. Furthermore, by marginalizing over latent discourse relations at test time, we obtain a discourse informed language model, which improves over a strong LSTM baseline. 1 Introduction Natural language processing (NLP) has recently ex- perienced a neural network “tsunami” (Manning, 2016). A key advantage of these neural architec- tures is that they employ discriminatively-trained distributed representations, which can capture the meaning of linguistic phenomena ranging from in- dividual words (Turian et al., 2010) to longer-range linguistic contexts at the sentence level (Socher et al., 2013) and beyond (Le and Mikolov, 2014). Be- cause they are discriminatively trained, these meth- ods can learn representations that yield very accurate predictive models (e.g., Dyer et al, 2015). However, in comparison with the probabilistic graphical models that were previously the dominant machine learning approach for NLP, neural archi- tectures lack flexibility. By treating linguistic an- notations as random variables, probabilistic graphi- cal models can marginalize over annotations that are unavailable at test or training time, elegantly model- ing multiple linguistic phenomena in a joint frame- work (Finkel et al., 2006). But because these graph- ical models represent uncertainty for every element in the model, adding too many layers of latent vari- ables makes them difficult to train. In this paper, we present a hybrid architecture that combines a recurrent neural network language model with a latent variable model over shallow discourse structure. In this way, the model learns a discriminatively-trained distributed representation of the local contextual features that drive word choice at the intra-sentence level, using techniques that are now state-of-the-art in language model- ing (Mikolov et al., 2010). However, the model treats shallow discourse structure — specifically, the relationships between pairs of adjacent sentences — as a latent variable. As a result, the model can act as both a discourse relation classifier and a language model. Specifically: • If trained to maximize the conditional like- lihood of the discourse relations, it outper- forms state-of-the-art methods for both im- plicit …

Content truncated to 500 words