Deep Pyramid Convolutional Neural Networks for Text Categorization

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 562–570 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1052 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 562–570 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1052 Deep Pyramid Convolutional Neural Networks for Text Categorization Rie Johnson RJ Research Consulting Tarrytown, NY, USA riejohnson@gmail.com Tong Zhang Tencent AI Lab Shenzhen, China bradymzhang@tencent.com Abstract This paper proposes a low-complexity word-level deep convolutional neural net- work (CNN) architecture for text catego- rization that can efficiently represent long- range associations in text. In the literature, several deep and complex neural networks have been proposed for this task, assum- ing availability of relatively large amounts of training data. However, the associated computational complexity increases as the networks go deeper, which poses serious challenges in practical applications. More- over, it was shown recently that shallow word-level CNNs are more accurate and much faster than the state-of-the-art very deep nets such as character-level CNNs even in the setting of large training data. Motivated by these findings, we carefully studied deepening of word-level CNNs to capture global representations of text, and found a simple network architecture with which the best accuracy can be obtained by increasing the network depth without increasing computational cost by much. We call it deep pyramid CNN. The pro- posed model with 15 weight layers out- performs the previous best models on six benchmark datasets for sentiment classifi- cation and topic categorization. 1 Introduction Text categorization is an important task whose ap- plications include spam detection, sentiment clas- sification, and topic classification. In recent years, neural networks that can make use of word or- der have been shown to be effective for text cat- egorization. While simple and shallow convolu- tional neural networks (CNNs) (Kim, 2014; John- son and Zhang, 2015a) were proposed for this task earlier, more recently, deep and more com- plex neural networks have also been studied, as- suming availability of relatively large amounts of training data (e.g., one million documents). Exam- ples are deep character-level CNNs (Zhang et al., 2015; Conneau et al., 2016), a complex combi- nation of CNNs and recurrent neural networks (RNNs) (Tang et al., 2015), and RNNs in a word- sentence hierarchy (Yang et al., 2016). A CNN is a feedforward network with convo- lution layers interleaved with pooling layers. Es- sentially, a convolution layer converts to a vector every small patch of data (either the original data such as text or image or the output of the pre- vious layer) at every location (e.g., 3-word win- dows around every word), which can be processed in parallel. By contrast, an RNN has connec- tions that form a cycle. In its typical application to text, a recurrent unit takes words one by one as well as its own output on the previous word, which is parallel-processing unfriendly. While both CNNs and RNNs can take advantage of word …

Content truncated to 500 words