Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations

Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations Eliyahu Kiperwasser Computer Science Department Bar-Ilan University Ramat-Gan, Israel elikip@gmail.com Yoav Goldberg Computer Science Department Bar-Ilan University Ramat-Gan, Israel yoav.goldberg@gmail.com Abstract We present a simple and effective scheme for dependency parsing which is based on bidirectional-LSTMs (BiLSTMs). Each sen- tence token is associated with a BiLSTM vec- tor representing the token in its sentential con- text, and feature vectors are constructed by concatenating a few BiLSTM vectors. The BiLSTM is trained jointly with the parser ob- jective, resulting in very effective feature ex- tractors for parsing. We demonstrate the ef- fectiveness of the approach by applying it to a greedy transition-based parser as well as to a globally optimized graph-based parser. The resulting parsers have very simple architec- tures, and match or surpass the state-of-the-art accuracies on English and Chinese. 1 Introduction The focus of this paper is on feature represen- tation for dependency parsing, using recent tech- niques from the neural-networks (“deep learning”) literature. Modern approaches to dependency pars- ing can be broadly categorized into graph-based and transition-based parsers (Kübler et al., 2009). Graph-based parsers (McDonald, 2006) treat pars- ing as a search-based structured prediction prob- lem in which the goal is learning a scoring func- tion over dependency trees such that the correct tree is scored above all other trees. Transition-based parsers (Nivre, 2004; Nivre, 2008) treat parsing as a sequence of actions that produce a parse tree, and a classifier is trained to score the possible actions at each stage of the process and guide the parsing pro- cess. Perhaps the simplest graph-based parsers are arc-factored (first order) models (McDonald, 2006), in which the scoring function for a tree decomposes over the individual arcs of the tree. More elaborate models look at larger (overlapping) parts, requiring more sophisticated inference and training algorithms (Martins et al., 2009; Koo and Collins, 2010). The basic transition-based parsers work in a greedy man- ner, performing a series of locally-optimal decisions, and boast very fast parsing speeds. More advanced transition-based parsers introduce some search into the process using a beam (Zhang and Clark, 2008) or dynamic programming (Huang and Sagae, 2010). Regardless of the details of the parsing frame- work being used, a crucial step in parser design is choosing the right feature function for the underly- ing statistical model. Recent work (see Section 2.2 for an overview) attempt to alleviate parts of the fea- ture function design problem by moving from lin- ear to non-linear models, enabling the modeler to focus on a small set of “core” features and leav- ing it up to the machine-learning machinery to come up with good feature combinations (Chen and Man- ning, 2014; Pei et al., 2015; Lei et al., 2014; Taub- Tabib et al., 2015). However, the need to carefully define a set of core features remains. For exam- ple, the work of Chen and Manning (2014) uses 18 different elements in its feature function, while the work of Pei et al. (2015) uses …

Content truncated to 500 words