Hierarchical Recurrent Neural Network for Document Modeling

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 899–907, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Hierarchical Recurrent Neural Network for Document Modeling Rui Lin†∗, Shujie Liu‡, Muyun Yang†, Mu Li‡, Ming Zhou‡, Sheng Li† †Harbin Institute of Technology {linrui,ymy}@mtlab.hit.edu.cn lisheng@hit.edu.cn ‡Microsoft Research {shujliu,muli,mingzhou}@microsoft.com Abstract This paper proposes a novel hierarchical recurrent neural network language model (HRNNLM) for document modeling. Af- ter establishing a RNN to capture the co- herence between sentences in a documen- t, HRNNLM integrates it as the sentence history information into the word level RNN to predict the word sequence with cross-sentence contextual information. A two-step training approach is designed, in which sentence-level and word-level lan- guage models are approximated for the convergence in a pipeline style. Examined by the standard sentence reordering sce- nario, HRNNLM is proved for its better accuracy in modeling the sentence coher- ence. And at the word level, experimen- tal results also indicate a significant low- er model perplexity, followed by a practi- cal better translation result when applied to a Chinese-English document translation reranking task. 1 Introduction Deep Neural Network (DNN), a neural network with multiple layers, has been proven powerful in many different domains, such as visual recogni- tion (Kavukcuoglu et al., 2010) and speech recog- nition (Dahl et al., 2012), ever since Hinton et al. (2006) formulated an efficient training method for it. In addition to the applications mentioned above, many neural network based methods have also been applied to natural language processing (NLP) tasks with great success. For example, Collobert et al. (2011) propose a generalized DNN framework for a variety of fundamental NLP tasks, including part-of-speech tagging (postag), chunking, named ∗Contribution during internship at Microsoft Research. entity recognition (NER), and semantic role label- ing. DNN is successfully introduced to do word- level language modeling, aka., to predict the next word given the history words. Bengio et al. (2003) propose a feedforward neural network to train a word-level language model with a limited n-gram history. To leverage as much history as possible, Mikolov et al. (2010) apply recurrent neural net- work to word-level language modeling. The mod- el absorbs one word each time, keeps the informa- tion in a history vector, and predicts the next word with all the word history in the vector. Word-level language model can only learn the relationship between words in one sentence. For sentences in one document which talks about one or several specific topics, the words in the next sentence are chosen partially in accordance with the previous sentences. To model this kind of co- herence of sentences, Le and Mikolov (2014) ex- tend word embedding learning network (Mikolov et al., 2013) to learn the paragraph embedding as a fixed-length vector representation for paragraph or sentence. Li and Hovy (2014) propose a neu- ral network coherence model which employs dis- tributed sentence representation and then predict the probability of whether a sequence of sentences is coherent or not. In contrast to the methods …

Content truncated to 500 words