Enhancing Sentence Embedding with Generalized Pooling

Proceedings of the 27th International Conference on Computational Linguistics, pages 1815–1826 Santa Fe, New Mexico, USA, August 20-26, 2018. 1815 Enhancing Sentence Embedding with Generalized Pooling Qian Chen University of Science and Technology of China cq1231@mail.ustc.edu.cn Zhen-Hua Ling University of Science and Technology of China zhling@ustc.edu.cn Xiaodan Zhu ECE, Queen’s University xiaodan.zhu@queensu.ca Abstract Pooling is an essential component of a wide variety of sentence representation and embedding models. This paper explores generalized pooling methods to enhance sentence embedding. We propose vector-based multi-head attention that includes the widely used max pooling, mean pool- ing, and scalar self-attention as special cases. The model benefits from properly designed penal- ization terms to reduce redundancy in multi-head attention. We evaluate the proposed model on three different tasks: natural language inference (NLI), author profiling, and sentiment clas- sification. The experiments show that the proposed model achieves significant improvement over strong sentence-encoding-based methods, resulting in state-of-the-art performances on four datasets. The proposed approach can be easily implemented for more problems than we discuss in this paper. 1 Introduction Distributed representation learned with neural networks has shown to be effective in modeling natural language at different granularities. Learning representation for words (Bengio et al., 2000; Mikolov et al., 2013; Pennington et al., 2014), for example, has achieved notable success. Much remains to be done to model larger spans of text such as sentences or documents. The approaches to computing sentence embedding generally fall into two categories. The first consists of learning sentence embedding with unsupervised learning, e.g., auto-encoder-based models (Socher et al., 2011), Paragraph Vector (Le and Mikolov, 2014), SkipThought vectors (Kiros et al., 2015), FastSent (Hill et al., 2016), among others. The second category consists of models trained with supervised learning, such as convolution neural networks (CNN) (Kim, 2014; Kalchbrenner et al., 2014), recurrent neural networks (RNN) (Conneau et al., 2017; Bowman et al., 2015), and tree-structure recursive networks (Socher et al., 2013; Zhu et al., 2015; Tai et al., 2015), just to name a few. Pooling is an essential component of a wide variety of sentence representation and embedding models. For example, in recurrent-neural-network-based models, pooling is often used to aggregate hidden states at different time steps (i.e., words in a sentence) to obtain sentence embedding. Convolutional neural networks (CNN) also often uses max or mean pooling to obtain a fixed-size sentence embedding. In this paper we explore generalized pooling methods to enhance sentence embedding. Specifically, by extending scalar self-attention models such as those proposed in Lin et al. (2017), we propose vector- based multi-head attention, which includes the widely used max pooling, mean pooling, and scalar self- attention itself as special cases. On one hand, the proposed method allows for extracting different aspects of the sentence into multiple vector representations through the multi-head mechanism. On the other, it allows the models to focus on one of many possible interpretations of the words encoded in the context vector through the vector-based attention mechanism. In the proposed model we design penalization terms to reduce redundancy in …

Content truncated to 500 words