Discriminative Neural Sentence Modeling by Tree-Based Convolution

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2315–2325, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Discriminative Neural Sentence Modeling by Tree-Based Convolution Lili Mou∗, Hao Peng∗, Ge Li†, Yan Xu, Lu Zhang, Zhi Jin {doublepower.mou, penghao.pku}@gmail.com {lige, xuyan14, zhanglu, zhijin}@sei.pku.edu.cn Software Institute, Peking University, 100871, P. R. China Abstract This paper proposes a tree-based con- volutional neural network (TBCNN) for discriminative sentence modeling. Our model leverages either constituency trees or dependency trees of sentences. The tree-based convolution process extracts sentences structural features, which are then aggregated by max pooling. Such ar- chitecture allows short propagation paths between the output layer and underlying feature detectors, enabling effective struc- tural feature learning and extraction. We evaluate our models on two tasks: senti- ment analysis and question classification. In both experiments, TBCNN outperforms previous state-of-the-art results, including existing neural networks and dedicated feature/rule engineering. We also make efforts to visualize the tree-based convo- lution process, shedding light on how our models work. 1 Introduction Discriminative sentence modeling aims to capture sentence meanings, and classify sentences accord- ing to certain criteria (e.g., sentiment). It is related to various tasks of interest, and has attracted much attention in the NLP community (Allan et al., 2003; Su and Markert, 2008; Zhao et al., 2015). Feature engineering—for example, n-gram fea- tures (Cui et al., 2006), dependency subtree fea- tures (Nakagawa et al., 2010), or more dedicated ones (Silva et al., 2011)—can play an important role in modeling sentences. Kernel machines, e.g., SVM, are exploited in Moschitti (2006) and Re- ichartz et al. (2010) by specifying a certain mea- sure of similarity between sentences, without ex- plicit feature representation. ∗These authors contribute equally to this paper. †To whom correspondence should be addressed. Recent advances of neural networks bring new techniques in understanding natural languages, and have exhibited considerable potential. Bengio et al. (2003) and Mikolov et al. (2013) propose un- supervised approaches to learn word embeddings, mapping discrete words to real-valued vectors in a meaning space. Le and Mikolov (2014) ex- tend such approaches to learn sentences’ and para- graphs’ representations. Compared with human engineering, neural networks serve as a way of au- tomatic feature learning (Bengio et al., 2013). Two widely used neural sentence models are convolutional neural networks (CNNs) and recur- sive neural networks (RNNs). CNNs can extract words’ neighboring features effectively with short propagation paths, but they do not capture inher- ent sentence structures (e.g., parse trees). RNNs encode, to some extent, structural information by recursive semantic composition along a parse tree. However, they may have difficulties in learning deep dependencies because of long propagation paths (Erhan et al., 2009). (CNNs/RNNs and a variant, recurrent networks, will be reviewed in Section 2.) A curious question is whether we can com- bine the advantages of CNNs and RNNs, i.e., whether we can exploit sentence structures (like RNNs) effectively with short propagation paths (like CNNs). In this paper, we propose a novel neural ar- chitecture for discriminative sentence …

Content truncated to 500 words