Molding CNNs for text: non-linear, non-consecutive convolutions

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1565–1575, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Molding CNNs for text: non-linear, non-consecutive convolutions Tao Lei, Regina Barzilay, and Tommi Jaakkola Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology {taolei, regina, tommi}@csail.mit.edu Abstract The success of deep learning often de- rives from well-chosen operational build- ing blocks. In this work, we revise the temporal convolution operation in CNNs to better adapt it to text processing. In- stead of concatenating word representa- tions, we appeal to tensor algebra and use low-rank n-gram tensors to directly exploit interactions between words already at the convolution stage. Moreover, we extend the n-gram convolution to non-consecutive words to recognize patterns with interven- ing words. Through a combination of low- rank tensors, and pattern weighting, we can efficiently evaluate the resulting con- volution operation via dynamic program- ming. We test the resulting architecture on standard sentiment classification and news categorization tasks. Our model achieves state-of-the-art performance both in terms of accuracy and training speed. For in- stance, we obtain 51.2% accuracy on the fine-grained sentiment classification task.1 1 Introduction Deep learning methods and convolutional neural networks (CNNs) among them have become de facto top performing techniques across a range of NLP tasks such as sentiment classification, question-answering, and semantic parsing. As methods, they require only limited domain knowl- edge to reach respectable performance with in- creasing data and computation, yet permit easy architectural and operational variations so as to fine tune them to specific applications to reach top performance. Indeed, their success is often con- tingent on specific architectural and operational choices. 1Our code and data are available at https://github. com/taolei87/text_convnet CNNs for text applications make use of tem- poral convolution operators or filters. Similar to image processing, they are applied at multi- ple resolutions, interspersed with non-linearities and pooling. The convolution operation itself is a linear mapping over “n-gram vectors” obtained by concatenating consecutive word (or character) representations. We argue that this basic build- ing block can be improved in two important re- spects. First, the power of n-grams derives pre- cisely from multi-way interactions and these are clearly missed (initially) with linear operations on stacked n-gram vectors. Non-linear interactions within a local context have been shown to improve empirical performance in various tasks (Mitchell and Lapata, 2008; Kartsaklis et al., 2012; Socher et al., 2013). Second, many useful patterns are expressed as non-consecutive phrases, such as se- mantically close multi-word expressions (e.g.,“not that good”, “not nearly as good”). In typical CNNs, such expressions would have to come to- gether and emerge as useful patterns after several layers of processing. We propose to use a feature mapping operation based on tensor products instead of linear opera- tions on stacked vectors. This enables us to di- rectly tap into non-linear interactions between ad- jacent word feature vectors (Socher et al., 2013; Lei et al., 2014). To offset the accompanying parametric explosion we maintain a low-rank rep- resentation …

Content truncated to 500 words