Modularized Syntactic Neural Networks for Sentence Classification

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2786–2792, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2786 Modularized Syntactic Neural Networks for Sentence Classification Haiyan Wu, Ying Liu∗, Shaoyun Shi Tsinghua University, Beijing, China, 100084 wuhy17@mails.tsinghua.edu.cn Abstract This paper focuses on tree-based modeling for the sentence classification task. In exist- ing works, aggregating on a syntax tree usu- ally considers local information of sub-trees. In contrast, in addition to the local informa- tion, our proposed Modularized Syntactic Neu- ral Network (MSNN) utilizes the syntax cate- gory labels and takes advantage of the global context while modeling sub-trees. In MSNN, each node of a syntax tree is modeled by a label-related syntax module. Each syntax module aggregates the outputs of lower-level modules, and finally, the root module pro- vides the sentence representation. We de- sign a tree-parallel mini-batch strategy for ef- ficient training and predicting. Experimental results on four benchmark datasets show that our MSNN significantly outperforms previous state-of-the-art tree-based methods on the sen- tence classification task. 1 Introduction Text classification is an important and fundamen- tal problem in natural language processing (NLP). With the increasing spread of the Internet, there are numerous applications of classification of short texts with only one sentence, for example, classify- ing questions according to what product or which part of the product architecture the question re- gards, sentiment analysis of customer reviews or tweets, and fast category detection based on news titles. Different from document classification, in which there are more topic words and features of writing styles, a single sentence contains limited information. Thus, understanding the meaning of a sentence is vital. Although sequential models like long short term memory (LSTM) (Hochreiter and Schmidhuber, * Corresponding Author: yingliu@mail.tsinghua.edu.cn 1997) and gated recurrent units (GRU) (Cho et al., 2014b) have been widely used and provide excel- lent performances, it is hard for them to capture the syntactic information, which is essential for under- standing sentences (Linzen et al., 2016). To utilize the syntactic information, some works proposed models taking parse trees or dependency trees as inputs (Le and Zuidema, 2015; Teng and Zhang, 2017; Socher et al., 2013; Zhu et al., 2015; Bow- man et al., 2016). Previous researchers have em- pirically verified that these methods help to model sentences (Li et al., 2015). However, to improve the model efficiency and simplify the implemen- tation, these methods binarize trees (Wang et al., 2007; Huang, 2007) so that they can be traversed by recursive neural networks (RvNN) or as a se- quence by RNN. Although some models, such as Tree-LSTM (Tai et al., 2015; Looks et al., 2017; Ran and Zhong, 2019), theoretically support origi- nal parse trees, child nodes are simply summed, but the relationships among them are not modeled. The authors only conduct experiments on binary trees. Binary trees weaken the syntactic information and conceal the relationships among nodes at the same level or different levels. Latent tree is another way of modeling sentences, but it does not …

Content truncated to 500 words