Information Aggregation via Dynamic Routing for Sequence Encoding

Proceedings of the 27th International Conference on Computational Linguistics, pages 2742–2752 Santa Fe, New Mexico, USA, August 20-26, 2018. 2742 Information Aggregation via Dynamic Routing for Sequence Encoding Jingjing Gong, Xipeng Qiu∗, Shaojing Wang, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University {jjgong15, xpqiu, sjwang17, xjhuang}@fudan.edu.cn Abstract While much progress has been made in how to encode a text sequence into a sequence of vectors, less attention has been paid to how to aggregate these preceding vectors (outputs of RNN/CNN) into fixed-size encoding vector. Usually, a simple max or average pooling is used, which is a bottom-up and passive way of aggregation and lack of guidance by task information. In this paper, we propose an aggregation mechanism to obtain a fixed-size encoding with a dynamic routing policy. The dynamic routing policy is dynamically deciding that what and how much information need be transferred from each word to the final encoding of the text sequence. Fol- lowing the work of Capsule Network, we design two dynamic routing policies to aggregate the outputs of RNN/CNN encoding layer into a final encoding vector. Compared to the other aggre- gation methods, dynamic routing can refine the messages according to the state of final encoding vector. Experimental results on five text classification tasks show that our method outperforms other aggregating models by a significant margin. Related source code is released on our github page1. 1 Introduction Learning the distributed representation of text sequences, such as sentences or documents, is crucial to a wide range of important natural language processing applications. A primary challenge is how to encode the variable-length text sequence into a fixed-size vector, which should fully capture the semantics of text. Many successful text encoding methods usually contain three key steps: (1) converting each word in a text sequence into its embedding; (2) taking as input the sequence of word embeddings, and computing the context-aware representation for each word with a recurrent neural network (RNN) (Hochreiter and Schmidhuber, 1997; Chung et al., 2014) or convolutional neural network (CNN) (Collobert et al., 2011; Kim, 2014); (3) summarizing the sentence meaning into a fixed-size vector by an aggregation operation. Then, these models are trained by combining a downstream task in a supervised or unsupervised way. Currently, much attention is paid to the first two steps, while the aggregation step is less emphasized on. Some simple aggregation methods, such as max (or average) pooling, is used to sum the RNN hidden states or convolved vectors, computed in the previous step, into a single vector. This kind of methods aggregate information in a bottom-up and passive way and are lack of the guide of task information. Recently, several works employ self-attention mechanism (Lin et al., 2017; Yang et al., 2016) on top of the recurrent or convolutional encoding layer to replace simple pooling. A basic assumption is that the words (or even sentences) are not equally important. One or several task-specific context vectors are used to assign a …

Content truncated to 500 words