Attentive Pooling with Learnable Norms for Text Representation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2961–2970 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2961 Attentive Pooling with Learnable Norms for Text Representation Chuhan Wu†, Fangzhao Wu‡, Tao Qi†, Xiaohui Cui§, Yongfeng Huang† †Department of Electronic Engineering & BNRist, Tsinghua University, Beijing 100084, China ‡Microsoft Research Asia, Beijing 100080, China §School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China. {wuchuhan15, wufangzhao}@gmail.com qit16@mails.tsinghua.edu.cn xcui@whu.edu.cn yfhuang@tsinghua.edu.cn Abstract Pooling is an important technique for learning text representations in many neural NLP mod- els. In conventional pooling methods such as average, max and attentive pooling, text rep- resentations are weighted summations of the L1 or L∞ norm of input features. However, their pooling norms are always fixed and may not be optimal for learning accurate text rep- resentations in different tasks. In addition, in many popular pooling methods such as max and attentive pooling some features may be over-emphasized, while other useful ones are not fully exploited. In this paper, we propose an Attentive Pooling with Learnable Norms (APLN) approach for text representation. Dif- ferent from existing pooling methods that use a fixed pooling norm, we propose to learn the norm in an end-to-end manner to auto- matically find the optimal ones for text rep- resentation in different tasks. In addition, we propose two methods to ensure the numeri- cal stability of the model training. The first one is scale limiting, which re-scales the in- put to ensure non-negativity and alleviate the risk of exponential explosion. The second one is re-formulation, which decomposes the ex- ponent operation to avoid computing the real- valued powers of the input and further accel- erate the pooling operation. Experimental re- sults on four benchmark datasets show that our approach can effectively improve the perfor- mance of attentive pooling. 1 Introduction In recent years, neural network based methods are widely used in the natural language process- ing (NLP) field to learn text representations (Yang et al., 2016; Peters et al., 2018). In these methods, pooling is a core technique to build the text repre- sentation vector from a collection of input feature vectors by summarizing their information (Lai et al., 2015). Thus, an effective pooling method Sentiment Classification Average Pooling The movie is good, but not to my taste Max Pooling The movie is good, but not to my taste Attentive Pooling The movie is good, but not to my taste News Topic Classification Average Pooling Fire on Queensland Island Takes Heavy Toll on Wildlife Max Pooling Fire on Queensland Island Takes Heavy Toll on Wildlife Attentive Pooling Fire on Queensland Island Takes Heavy Toll on Wildlife Figure 1: The pooling weights of several different pool- ing methods on the representations produced by an LSTM network. Darker colors indicate higher weights. that can select salient features accurately will fa- cilitate many NLP methods (Ma et al., 2017). Among existing pooling methods, average pool- ing is a representative one which takes the aver- age of the L1 norm …

Content truncated to 500 words