Multichannel Variable-Size Convolution for Sentence Classification

Proceedings of the 19th Conference on Computational Language Learning, pages 204–214, Beijing, China, July 30-31, 2015. c⃝2015 Association for Computational Linguistics Multichannel Variable-Size Convolution for Sentence Classification Wenpeng Yin and Hinrich Sch¨utze Center for Information and Language Processing University of Munich, Germany wenpeng@cis.uni-muenchen.de Abstract We propose MVCNN, a convolution neu- ral network (CNN) architecture for sen- tence classification. It (i) combines di- verse versions of pretrained word embed- dings and (ii) extracts features of multi- granular phrases with variable-size convo- lution filters. We also show that pretrain- ing MVCNN is critical for good perfor- mance. MVCNN achieves state-of-the-art performance on four tasks: on small-scale binary, small-scale multi-class and large- scale Twitter sentiment prediction and on subjectivity classification. 1 Introduction Different sentence classification tasks are crucial for many Natural Language Processing (NLP) ap- plications. Natural language sentences have com- plicated structures, both sequential and hierarchi- cal, that are essential for understanding them. In addition, how to decode and compose the features of component units, including single words and variable-size phrases, is central to the sentence classification problem. In recent years, deep learning models have achieved remarkable results in computer vision (Krizhevsky et al., 2012), speech recognition (Graves et al., 2013) and NLP (Collobert and We- ston, 2008). A problem largely specific to NLP is how to detect features of linguistic units, how to conduct composition over variable-size sequences and how to use them for NLP tasks (Collobert et al., 2011; Kalchbrenner et al., 2014; Kim, 2014). Socher et al. (2011a) proposed recursive neural networks to form phrases based on parsing trees. This approach depends on the availability of a well performing parser; for many languages and do- mains, especially noisy domains, reliable parsing is difficult. Hence, convolution neural networks (CNN) are getting increasing attention, for they are able to model long-range dependencies in sen- tences via hierarchical structures (Dos Santos and Gatti, 2014; Kim, 2014; Denil et al., 2014). Cur- rent CNN systems usually implement a convolu- tion layer with fixed-size filters (i.e., feature detec- tors), in which the concrete filter size is a hyper- parameter. They essentially split a sentence into multiple sub-sentences by a sliding window, then determine the sentence label by using the domi- nant label across all sub-sentences. The underly- ing assumption is that the sub-sentence with that granularity is potentially good enough to represent the whole sentence. However, it is hard to find the granularity of a “good sub-sentence” that works well across sentences. This motivates us to imple- ment variable-size filters in a convolution layer in order to extract features of multigranular phrases. Breakthroughs of deep learning in NLP are also based on learning distributed word representations – also called “word embeddings” – by neural lan- guage models (Bengio et al., 2003; Mnih and Hin- ton, 2009; Mikolov et al., 2010; Mikolov, 2012; Mikolov et al., 2013a). Word embeddings are de- rived by projecting words from a sparse, 1-of-V encoding (V : vocabulary size) onto a lower di- mensional and dense vector space via hidden …

Content truncated to 500 words