MODE-LSTM: A Parameter-efficient Recurrent Network with Multi-Scale for Sentence Classification

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6705–6715, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6705 MODE-LSTM: A Parameter-efficient Recurrent Network with Multi-Scale for Sentence Classification Qianli Ma, Zhenxi Lin, Jiangyue Yan, Zipeng Chen, Liuhong Yu School of Computer Science and Engineering, South China University of Technology, Guangzhou, China qianlima@scut.edu.cn zhenxi lin@foxmail.com Abstract The central problem of sentence classification is to extract multi-scale n-gram features for understanding the semantic meaning of sen- tences. Most existing models tackle this prob- lem by stacking CNN and RNN models, which easily leads to feature redundancy and over- fitting because of relatively limited datasets. In this paper, we propose a simple yet ef- fective model called Multi-scale Orthogonal inDependEnt LSTM (MODE-LSTM), which not only has effective parameters and good generalization ability, but also considers multi- scale n-gram features. We disentangle the hid- den state of the LSTM into several indepen- dently updated small hidden states and apply an orthogonal constraint on their recurrent ma- trices. We then equip this structure with slid- ing windows of different sizes for extracting multi-scale n-gram features. Extensive exper- iments demonstrate that our model achieves better or competitive performance against state-of-the-art baselines on eight benchmark datasets. We also combine our model with BERT to further boost the generalization per- formance. 1 Introduction Sentence classification (SC) is a fundamental and traditional task in natural language processing (NLP), which is widely used in many subareas, such as sentiment analysis (Wang et al., 2016a, 2018) and question classification (Shi et al., 2016). The central problem of SC is to understand the se- mantic meaning of a sentence by some key-phrases located at different positions (Wang et al., 2015). CNNs excel at extracting n-gram features of sen- tences through a convolution operation followed by non-linear and pooling layers and have achieved impressive results in sentence classification (Kalch- brenner et al., 2014; Kim, 2014). However, the con- volution operation itself is linear, which may not be sufficient to model the non-consecutive dependency of the phrase (Lei et al., 2015) and may lose the sequential information (Madasu and Anvesh Rao, 2019). As shown in Figure 1, the weighted sum of the phrase “not almost as bad” does not capture the non-consecutive dependency of “not bad” very well and ignores the sequential information. Figure 1: An example with variable-size phrases. On the other hand, LSTMs (Hochreiter and Schmidhuber, 1997) are suitable for encoding structure-dependent semantics by storing previous word representations and preserving sequential in- formation. However, LSTMs are still biased toward later words and ignoring the earlier words (Yin et al., 2017), so some current methods (Lai et al., 2015; Wang et al., 2016b; Zhang et al., 2016a; Song et al., 2018) combine the CNN and LSTM by stack- ing. However, merely stacking multiple layers can easily lead to feature redundancy and overfitting, because only relatively small training sets are avail- able for SC tasks (Yin and Sch¨utze, 2015; Guo et al., 2019). Hence, some researchers (Zhao et al., …

Content truncated to 500 words