A Re-ranking Model for Dependency Parser with Recursive Convolutional Neural Network

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1159–1168, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics A Re-ranking Model for Dependency Parser with Recursive Convolutional Neural Network Chenxi Zhu, Xipeng Qiu∗, Xinchi Chen, Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University 825 Zhangheng Road, Shanghai, China {czhu13,xpqiu,xinchichen13,xjhuang}@fudan.edu.cn Abstract In this work, we address the prob- lem to model all the nodes (words or phrases) in a dependency tree with the dense representations. We propose a recursive convolutional neural network (RCNN) architecture to capture syntac- tic and compositional-semantic represen- tations of phrases and words in a depen- dency tree. Different with the original re- cursive neural network, we introduce the convolution and pooling layers, which can model a variety of compositions by the feature maps and choose the most infor- mative compositions by the pooling lay- ers. Based on RCNN, we use a discrimina- tive model to re-rank a k-best list of can- didate dependency parsing trees. The ex- periments show that RCNN is very effec- tive to improve the state-of-the-art depen- dency parsing on both English and Chi- nese datasets. 1 Introduction Feature-based discriminative supervised models have achieved much progress in dependency pars- ing (Nivre, 2004; Yamada and Matsumoto, 2003; McDonald et al., 2005), which typically use mil- lions of discrete binary features generated from a limited size training data. However, the ability of these models is restricted by the design of features. The number of features could be so large that the result models are too complicated for practical use and prone to overfit on training corpus due to data sparseness. Recently, many methods are proposed to learn various distributed representations on both syn- tax and semantics levels. These distributed repre- sentations have been extensively applied on many ∗Corresponding author. a,Det red,JJ bike,NN Convolution Pooling a red bike,NN a bike,NN red bike,NN Figure 1: Illustration of a RCNN unit. natural language processing (NLP) tasks, such as syntax (Turian et al., 2010; Mikolov et al., 2010; Collobert et al., 2011; Chen and Manning, 2014) and semantics (Huang et al., 2012; Mikolov et al., 2013). Distributed representations are to represent words (or phrase) by the dense, low-dimensional and real-valued vectors, which help address the curse of dimensionality and have better general- ization than discrete representations. For dependency parsing, Chen et al. (2014) and Bansal et al. (2014) used the dense vectors (embeddings) to represent words or features and found these representations are complementary to the traditional discrete feature representation. However, these two methods only focus on the dense representations (embeddings) of words or features. These embeddings are pre-trained and keep unchanged in the training phase of parsing model, which cannot be optimized for the specific tasks. Besides, it is also important to represent the (unseen) phrases with dense vector in dependency parsing. Since the dependency tree is also in re- cursive structure, it is …

Content truncated to 500 words