Improved Transition-Based Parsing and Tagging with Neural Networks

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1354–1359, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Improved Transition-Based Parsing and Tagging with Neural Networks Chris Alberti David Weiss Greg Coppola Slav Petrov Google Inc New York, NY {chrisalberti,djweiss,gcoppola,slav}@google.com Abstract We extend and improve upon recent work in struc- tured training for neural network transition-based dependency parsing. We do this by experimenting with novel features, additional transition systems and by testing on a wider array of languages. In par- ticular, we introduce set-valued features to encode the predicted morphological properties and part-of- speech confusion sets of the words being parsed. We also investigate the use of joint parsing and part- of-speech tagging in the neural paradigm. Finally, we conduct a multi-lingual evaluation that demon- strates the robustness of the overall structured neu- ral approach, as well as the benefits of the exten- sions proposed in this work. Our research further demonstrates the breadth of the applicability of neu- ral network methods to dependency parsing, as well as the ease with which new features can be added to neural parsing models. 1 Introduction Transition-based parsers (Nivre, 2008) are ex- tremely popular because of their high accuracy and speed. Inspired by the greedy neural net- work transition-based parser of Chen and Man- ning (2014), Weiss et al. (2015) and Zhou et al. (2015) concurrently developed structured neural network parsers that use beam search and achieve state-of-the-art accuracies for English dependency parsing.1 While very successful, these parsers have made use only of a small fraction of the rich options provided inside the transition-based framework: for example, all of these parsers use virtually identical atomic features and the arc- standard transition system. In this paper we extend this line of work and introduce two new types of features that sig- nificantly improve parsing performance: (1) a set-valued (i.e., bag-of-words style) feature for 1There is of course a much longer tradition of neural net- work dependency parsing models, going back at least to Titov and Henderson (2007). each word’s morphological attributes, and (2) a weighted set-valued feature for each word’s k-best POS tags. These features can be integrated nat- urally as atomic inputs to the embedding layer of the network and the model can learn arbitrary con- junctions with all other features through the hid- den layers. In contrast, integrating such features into a model with discrete features requires non- trivial manual tweaking. For example, Bohnet and Nivre (2012) had to carefully discretize the real-valued POS tag score in order to combine it with the other discrete binary features in their sys- tem. Additionally, we also experiment with differ- ent transition systems, most notably the integrated parsing and part-of-speech (POS) tagging system of Bohnet and Nivre (2012) and also the swap sys- tem of Nivre (2009). We evaluate our parser on the CoNLL ’09 shared task dependency treebanks, as well as on two English setups, achieving the best published numbers in many cases. 2 Model In this section, …

Content truncated to 500 words