Probabilistic Graph-based Dependency Parsing with Convolutional Neural Network

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1382–1392, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Probabilistic Graph-based Dependency Parsing with Convolutional Neural Network Zhisong Zhang1,2, Hai Zhao1,2,∗, Lianhui Qin1,2 1Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China 2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China {zzs2011,qinlianhui}@sjtu.edu.cn,zhaohai@cs.sjtu.edu.cn Abstract This paper presents neural probabilistic parsing models which explore up to third- order graph-based parsing with maximum likelihood training criteria. Two neural network extensions are exploited for per- formance improvement. Firstly, a convo- lutional layer that absorbs the influences of all words in a sentence is used so that sentence-level information can be effec- tively captured. Secondly, a linear layer is added to integrate different order neu- ral models and trained with perceptron method. The proposed parsers are evalu- ated on English and Chinese Penn Tree- banks and obtain competitive accuracies. 1 Introduction Neural network methods have shown great promise in the field of parsing and other related natural language processing tasks, exploiting more complex features with distributed representation and non-linear neural network (Wang et al., 2013; Wang et al., 2014; Cai and Zhao, 2016; Wang et al., 2016). In transition-based dependency pars- ing, neural models that can represent the partial or whole parsing histories have been explored (Weiss et al., 2015; Dyer et al., 2015). While for graph- based parsing, on which we focus in this work, Pei et al. (2015) also show the effectiveness of neural methods. ∗Corresponding author. This paper was partially sup- ported by Cai Yuanpei Program (CSC No. 201304490199 and No. 201304490171), National Natural Science Founda- tion of China (No. 61170114 and No. 61272248), National Basic Research Program of China (No. 2013CB329401), Major Basic Research Program of Shanghai Science and Technology Committee (No. 15JC1400103), Art and Sci- ence Interdisciplinary Funds of Shanghai Jiao Tong Univer- sity (No. 14JCRZ04), and Key Project of National Society Science Foundation of China (No. 15-ZDA041). The graph-based parser generally consists of two components: one is the parsing algorithm for inference or searching the most likely parse tree, the other is the parameter estimation approach for the machine learning models. For the former, clas- sical dynamic programming algorithms are usu- ally adopted, while for the latter, there are vari- ous solutions. Like some previous neural methods (Socher et al., 2010; Socher et al., 2013), to tackle the structure prediction problems, Pei et al. (2015) utilize a max-margin training criterion, which does not include probabilistic explanations. Re-visiting the traditional probabilistic criteria in log-linear models, this work utilizes maximum likelihood for neural network training. Durrett and Klein (2015) adopt this method for constituency pars- ing, which scores the anchored rules with neu- ral models and formalizes the probabilities with tree-structured random fields. Motivated by this work, we utilize the probabilistic treatment for de- pendency parsing: scoring the edges or high-order sub-trees with a neural model and calculating the gradients according to …

Content truncated to 500 words