Evaluating a Deterministic Shift-Reduce Neural Parser for Constituent Parsing

Evaluating a Deterministic Shift-Reduce Neural Parser for Constituent Parsing Hao Zhou†, Yue Zhang‡, Shujian Huang†, Xinyu Dai†, Jiajun Chen† †State Key Laboratory for Novel Software Technology, Nanjing University, China, ‡Singapore University of Technology and Design, Singapore {zhouh, huangsj, daixy, chenjj}@nlp.nju.edu.cn, yue zhang@sutd.edu.sg Abstract Greedy transition-based parsers are appealing for their very fast speed, with reasonably high accuracies. In this paper, we build a fast shift-reduce neural constituent parser by using a neural network to make local decisions. One challenge to the parsing speed is the large hidden and output layer sizes caused by the number of constituent labels and branching options. We speed up the parser by using a hierarchical output layer, inspired by the hierarchical log-bilinear neural language model. In standard WSJ experiments, the neural parser achieves an almost 2.4 time speed up (320 sen/sec) compared to a non-hierarchical baseline without significant accuracy loss (89.06 vs 89.13 F-score). Keywords: Hierarchical Model, Neural Networks, Parsing 1. Introduction With the growth of demands for processing massive amounts of web data, the speed of NLP systems has be- come a key practical performance. Transition-based parsers build outputs by using a sequence of shift-reduce actions (Sagae and Lavie, 2005; Zhang and Nivre, 2011; Bohnet and Nivre, 2012; Zhou et al., 2015). They are attractive by their fast speed, offering linear time complexity with deterministic decoding algorithms. Linear classifiers have traditionally been used to build very fast deterministic parsers (Nivre, 2008; Goldberg and Nivre, 2013). Recently, Chen and Manning (2014) proposed a transition- based dependency parser, which adopts a neural network as the classifier. It has two advantages compared to the tra- ditional linear model (Nivre, 2008), namely fast speed and automatic feature combination. In particular, the speed of a neural dependency parser can be 1000 sentences per second on a single CPU, which is significant faster than traditional parsers. In this paper, we explore the method of Chen and Manning (2014) for fast deterministic transition-based constituent parsing. This is a more challenging task compared with neural dependency parsing (Table 1). First, due to unary actions and temporary nodes caused by binarization, the number of actions in a transition-based constituent parser is much larger than in a dependency parser, resulting in a large output layer for the neural network. Second, we find empirically that the neural constituent parser needs a much larger hidden layer (i.e. 800) to obtain reasonable perfor- mance compared to the neural dependency parser (i.e. 200). As a result, the speed of neural constituent parser is much slower than the dependency counterpart. Motivated by hierarchical log-bilinear neural language models (Morin and Bengio, 2005; Mnih and Hinton, 2009), we propose a hierarchical-output neural model to speed up the parser. A hierarchical neural language model divides a huge vocabulary into a hierarchy of clusters, and con- structs a binary tree upon the word clusters. Each word corresponds to a leaf node, and can be uniquely specified by the path from the root. Hidden Action Computation Dependency 200 27 5,400 Constituent 800 131 104,800 …

Content truncated to 500 words