Sentence Modeling with Gated Recursive Neural Network

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 793–798, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Sentence Modeling with Gated Recursive Neural Network Xinchi Chen, Xipeng Qiu∗, Chenxi Zhu, Shiyu Wu, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {xinchichen13,xpqiu,czhu13,syu13,xjhuang}@fudan.edu.cn Abstract Recently, neural network based sentence modeling methods have achieved great progress. Among these methods, the re- cursive neural networks (RecNNs) can ef- fectively model the combination of the words in sentence. However, RecNNs need a given external topological struc- ture, like syntactic tree. In this paper, we propose a gated recursive neural net- work (GRNN) to model sentences, which employs a full binary tree (FBT) struc- ture to control the combinations in re- cursive structure. By introducing two kinds of gates, our model can better model the complicated combinations of features. Experiments on three text classification datasets show the effectiveness of our model. 1 Introduction Recently, neural network based sentence modeling approaches have been increasingly focused on for their ability to minimize the efforts in feature en- gineering, such as Neural Bag-of-Words (NBoW), Recurrent Neural Network (RNN) (Mikolov et al., 2010), Recursive Neural Network (RecNN) (Pol- lack, 1990; Socher et al., 2013b; Socher et al., 2012) and Convolutional Neural Network (CNN) (Kalchbrenner et al., 2014; Hu et al., 2014). Among these methods, recursive neural net- works (RecNNs) have shown their excellent abil- ities to model the word combinations in sentence. However, RecNNs require a pre-defined topolog- ical structure, like parse tree, to encode sentence, which limits the scope of its application. Cho et al. (2014) proposed the gated recursive convolutional neural network (grConv) by utilizing the directed acyclic graph (DAG) structure instead of parse tree ∗Corresponding author. cannot agree with you I more agree with you I more cannot Figure 1: Example of Gated Recursive Neural Networks (GRNNs). Left is a GRNN using a di- rected acyclic graph (DAG) structure. Right is a GRNN using a full binary tree (FBT) structure. (The green nodes, gray nodes and white nodes illustrate the positive, negative and neutral senti- ments respectively.) to model sentences. However, DAG structure is relatively complicated. The number of the hidden neurons quadraticly increases with the length of sentences so that grConv cannot effectively deal with long sentences. Inspired by grConv, we propose a gated recur- sive neural network (GRNN) for sentence model- ing. Different with grConv, we use the full binary tree (FBT) as the topological structure to recur- sively model the word combinations, as shown in Figure 1. The number of the hidden neurons lin- early increases with the length of sentences. An- other difference is that we introduce two kinds of gates, reset and update gates (Chung et al., 2014), to control the combinations in recursive structure. With these two gating mechanisms, our model can better model the complicated combinations of fea- tures and capture the long dependency interac- tions. In our previous …

Content truncated to 500 words