Why and when should you pool? Analyzing Pooling in Recurrent Architectures

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4568–4586 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4568 Why and when should you pool? Analyzing Pooling in Recurrent Architectures Pratyush Maini†, Keshav Kolluru†, Danish Pruthi‡, Mausam† †Indian Institute of Technology, Delhi, India ‡Carnegie Mellon University, Pittsburgh, USA {pratyush.maini, keshav.kolluru}@gmail.com, ddanish@cs.cmu.edu, mausam@cse.iitd.ac.in Abstract Pooling-based recurrent neural architectures consistently outperform their counterparts without pooling on sequence classification tasks. However, the reasons for their enhanced performance are largely unexamined. In this work, we explore three commonly used pool- ing techniques (mean-pooling, max-pooling, and attention1), and propose max-attention, a novel variant that captures interactions among predictive tokens in a sentence. Using novel experiments, we demonstrate that pooling ar- chitectures substantially differ from their non- pooling equivalents in their learning ability and positional biases: (i) pooling facilitates better gradient flow than BiLSTMs in initial training epochs, and (ii) BiLSTMs are biased towards tokens at the beginning and end of the input, whereas pooling alleviates this bias. Consequently, we find that pooling yields large gains in low resource scenarios, and instances when salient words lie towards the middle of the input. Across several text classification tasks, we find max-attention to frequently out- perform other pooling techniques.2 1 Introduction Pooling mechanisms are ubiquitous components in Recurrent Neural Networks (RNNs) used for natural language tasks. Pooling operations consoli- date hidden representations from RNNs into a sin- gle sentence representation. Various pooling tech- niques, like mean-pooling, max-pooling, and atten- tion, have been shown to improve the performance of RNNs on text classification tasks (Lai et al., 2015; Conneau et al., 2017). Despite widespread adoption, precisely how and when pooling benefits the models is largely under-explored. 1Attention aggregates representations via a weighted sum, thus we consider it under the umbrella of pooling in this paper. 2Code and data is made available at https://github.com/dair- iitd/PoolingAnalysis. In this work, we perform an in-depth analysis comparing popular pooling methods, and proposed max-attention, with standard BiLSTMs for several text classification tasks. We identify two key fac- tors that explain the benefits of pooling techniques: learnability, and positional invariance. First, we analyze the flow of gradients for differ- ent classification tasks to assess the learning ability of BiLSTMs (§ 5). We observe that the gradients corresponding to hidden representations in the mid- dle of the sequence vanish during the initial epochs. On training for more examples, these gradients slowly recover, suggesting that the gates of stan- dard BiLSTMs require many examples to learn. In contrast, we find the gradient norms in pooling- based architectures to be free from this problem. Pooling enables a fraction of the gradients to di- rectly reach any hidden state instead of having to backpropagate through a long series of recurrent cells. Thus we hypothesize, and subsequently con- firm, that pooling is particularly beneficial for tasks with long input sequences. Second, we explore the positional biases of BiL- STMs, with and without pooling (§ 6). Across several classification tasks, and various novel ex- perimental setups, we expose …

Content truncated to 500 words