Dependency Sensitive Convolutional Neural Networks for Modeling Sentences and Documents

Proceedings of NAACL-HLT 2016, pages 1512–1521, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Dependency Sensitive Convolutional Neural Networks for Modeling Sentences and Documents Rui Zhang Department of EECS University of Michigan Ann Arbor, MI, USA ryanzh@umich.edu Honglak Lee Department of EECS University of Michigan Ann Arbor, MI, USA honglak@eecs.umich.edu Dragomir Radev Department of EECS and School of Information University of Michigan Ann Arbor, MI, USA radev@umich.edu Abstract The goal of sentence and document model- ing is to accurately represent the meaning of sentences and documents for various Natural Language Processing tasks. In this work, we present Dependency Sensitive Convolutional Neural Networks (DSCNN) as a general- purpose classification system for both sen- tences and documents. DSCNN hierarchically builds textual representations by processing pretrained word embeddings via Long Short- Term Memory networks and subsequently ex- tracting features with convolution operators. Compared with existing recursive neural mod- els with tree structures, DSCNN does not rely on parsers and expensive phrase label- ing, and thus is not restricted to sentence- level tasks. Moreover, unlike other CNN- based models that analyze sentences locally by sliding windows, our system captures both the dependency information within each sen- tence and relationships across sentences in the same document. Experiment results demon- strate that our approach is achieving state-of- the-art performance on several tasks, includ- ing sentiment analysis, question type classifi- cation, and subjectivity classification. 1 Introduction Sentence and document modeling systems are im- portant for many Natural Language Processing (NLP) applications. The challenge for textual mod- eling is to capture features for different text units and to perform compositions over variable-length sequences (e.g., phrases, sentences, documents). As a traditional method, the bag-of-words model treats sentences and documents as unordered collections of words. In this way, however, the bag-of-words model fails to encode word orders and syntactic structures. By contrast, order-sensitive models based on neu- ral networks are becoming increasingly popular thanks to their ability to capture word order infor- mation. Many prevalent order-sensitive neural mod- els can be categorized into two classes: Recursive models and Convolutional Neural Networks (CNN) models. Recursive models can be considered as gen- eralizations of traditional sequence-modeling neural networks to tree structures. For example, (Socher et al., 2013) uses Recursive Neural Networks to build representations of phrases and sentences by combin- ing neighboring constituents based on the parse tree. In their model, the composition is performed in a bottom-up way from leaf nodes of tokens until the root node of the parsing tree is reached. CNN based models, as the second category, utilize convolutional filters to extract local features (Kalchbrenner et al., 2014; Kim, 2014) over embedding matrices consist- ing of pretrained word vectors. Therefore, the model actually splits the sentence locally into n-grams by sliding windows. However, despite their ability to account for word orders, order-sensitive models based on neu- ral networks still suffer from several disadvantages. First, recursive models depend on well-performing parsers, which can be difficult for many languages or noisy domains (Iyyer et al., 2015; Ma …

Content truncated to 500 words