Structured Training for Neural Network Transition-Based Parsing

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 323–333, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Structured Training for Neural Network Transition-Based Parsing David Weiss Chris Alberti Michael Collins Slav Petrov Google Inc New York, NY {djweiss,chrisalberti,mjcollins,slav}@google.com Abstract We present structured perceptron training for neural network transition-based dependency parsing. We learn the neural network representation using a gold corpus augmented by a large number of automat- ically parsed sentences. Given this fixed network representation, we learn a final layer using the struc- tured perceptron with beam-search decoding. On the Penn Treebank, our parser reaches 94.26% un- labeled and 92.41% labeled attachment accuracy, which to our knowledge is the best accuracy on Stanford Dependencies to date. We also provide in- depth ablative analysis to determine which aspects of our model provide the largest gains in accuracy. 1 Introduction Syntactic analysis is a central problem in lan- guage understanding that has received a tremen- dous amount of attention. Lately, dependency parsing has emerged as a popular approach to this problem due to the availability of dependency tree- banks in many languages (Buchholz and Marsi, 2006; Nivre et al., 2007; McDonald et al., 2013) and the efficiency of dependency parsers. Transition-based parsers (Nivre, 2008) have been shown to provide a good balance between efficiency and accuracy. In transition-based pars- ing, sentences are processed in a linear left to right pass; at each position, the parser needs to choose from a set of possible actions defined by the transition strategy. In greedy models, a classi- fier is used to independently decide which transi- tion to take based on local features of the current parse configuration. This classifier typically uses hand-engineered features and is trained on indi- vidual transitions extracted from the gold transi- tion sequence. While extremely fast, these greedy models typically suffer from search errors due to the inability to recover from incorrect decisions. Zhang and Clark (2008) showed that a beam- search decoding algorithm utilizing the structured perceptron training algorithm can greatly improve accuracy. Nonetheless, significant manual fea- ture engineering was required before transition- based systems provided competitive accuracy with graph-based parsers (Zhang and Nivre, 2011), and only by incorporating graph-based scoring func- tions were Bohnet and Kuhn (2012) able to exceed the accuracy of graph-based approaches. In contrast to these carefully hand-tuned ap- proaches, Chen and Manning (2014) recently presented a neural network version of a greedy transition-based parser. In their model, a feed- forward neural network with a hidden layer is used to make the transition decisions. The hidden layer has the power to learn arbitrary combinations of the atomic inputs, thereby eliminating the need for hand-engineered features. Furthermore, because the neural network uses a distributed representa- tion, it is able to model lexical, part-of-speech (POS) tag, and arc label similarities in a contin- uous space. However, although their model out- performs its greedy hand-engineered counterparts, it is not competitive with state-of-the-art …

Content truncated to 500 words