Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3485–3495, Osaka, Japan, December 11-17 2016. Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling Peng Zhou1, Zhenyu Qi1∗, Suncong Zheng1, Jiaming Xu1, Hongyun Bao1, Bo Xu1,2 (1) Institute of Automation, Chinese Academy of Sciences (2) Center for Excellence in Brain Science and Intelligence Technology {zhoupeng2013, zhenyu.qi, zhengsuncong, jiaming.xu, hongyun.bao, xubo}@ia.ac.cn Abstract Recurrent Neural Network (RNN) is one of the most popular architectures used in Natural Lan- guage Processsing (NLP) tasks because its recurrent structure is very suitable to process variable- length text. RNN can utilize distributed representations of words by first converting the tokens comprising each text into vectors, which form a matrix. And this matrix includes two dimen- sions: the time-step dimension and the feature vector dimension. Then most existing models usually utilize one-dimensional (1D) max pooling operation or attention-based operation only on the time-step dimension to obtain a fixed-length vector. However, the features on the feature vector dimension are not mutually independent, and simply applying 1D pooling operation over the time-step dimension independently may destroy the structure of the feature representation. On the other hand, applying two-dimensional (2D) pooling operation over the two dimensions may sample more meaningful features for sequence modeling tasks. To integrate the features on both dimensions of the matrix, this paper explores applying 2D max pooling operation to obtain a fixed-length representation of the text. This paper also utilizes 2D convolution to sample more meaningful information of the matrix. Experiments are conducted on six text classification tasks, including sentiment analysis, question classification, subjectivity classification and newsgroup classification. Compared with the state-of-the-art models, the proposed models achieve excellent performance on 4 out of 6 tasks. Specifically, one of the proposed models achieves highest accu- racy on Stanford Sentiment Treebank binary classification and fine-grained classification tasks. 1 Introduction Text classification is an essential component in many NLP applications, such as sentiment analysis (Socher et al., 2013), relation extraction (Zeng et al., 2014) and spam detection (Wang, 2010). There- fore, it has attracted considerable attention from many researchers, and various types of models have been proposed. As a traditional method, the bag-of-words (BoW) model treats texts as unordered sets of words (Wang and Manning, 2012). In this way, however, it fails to encode word order and syntactic feature. Recently, order-sensitive models based on neural networks have achieved tremendous success for text classification, and shown more significant progress compared with BoW models. The challenge for textual modeling is how to capture features for different text units, such as phrases, sentences and documents. Benefiting from its recurrent structure, RNN, as an alternative type of neural networks, is very suitable to process the variable-length text. RNN can capitalize on distributed representations of words by first converting the tokens compris- ing each text into vectors, which form a matrix. This matrix includes two dimensions: the time-step dimension and the feature vector dimension, and it will be updated in the process of learning feature representation. …

Content truncated to 500 words