Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 349–359, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs Miguel Ballesteros♦♠ Chris Dyer♣♠ Noah A. Smith♥ ♦NLP Group, Pompeu Fabra University, Barcelona, Spain ♠School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA ♣Marianas Labs, Pittsburgh, PA, USA ♥Computer Science & Engineering, University of Washington, Seattle, WA, USA miguel.ballesteros@upf.edu, chris@marianaslabs.com, nasmith@cs.washington.edu Abstract We present extensions to a continuous- state dependency parsing method that makes it applicable to morphologically rich languages. Starting with a high- performance transition-based parser that uses long short-term memory (LSTM) re- current neural networks to learn repre- sentations of the parser state, we replace lookup-based word representations with representations constructed from the or- thographic representations of the words, also using LSTMs. This allows statistical sharing across word forms that are simi- lar on the surface. Experiments for mor- phologically rich languages show that the parsing model benefits from incorporating the character-based encodings of words. 1 Introduction At the heart of natural language parsing is the chal- lenge of representing the “state” of an algorithm— what parts of a parse have been built and what parts of the input string are not yet accounted for— as it incrementally constructs a parse. Traditional approaches rely on independence assumptions, de- composition of scoring functions, and/or greedy approximations to keep this space manageable. Continuous-state parsers have been proposed, in which the state is embedded as a vector (Titov and Henderson, 2007; Stenetorp, 2013; Chen and Manning, 2014; Dyer et al., 2015; Zhou et al., 2015; Weiss et al., 2015). Dyer et al. reported state-of-the-art performance on English and Chi- nese benchmarks using a transition-based parser whose continuous-state embeddings were con- structed using LSTM recurrent neural networks (RNNs) whose parameters were estimated to max- imize the probability of a gold-standard sequence of parse actions. The primary contribution made in this work is to take the idea of continuous-state parsing a step fur- ther by making the word embeddings that are used to construct the parse state sensitive to the mor- phology of the words.1 Since it it is well known that a word’s form often provides strong evidence regarding its grammatical role in morphologically rich languages (Ballesteros, 2013, inter alia), this has promise to improve accuracy and statistical ef- ficiency relative to traditional approaches that treat each word type as opaque and independently mod- eled. In the traditional parameterization, words with similar grammatical roles will only be em- bedded near each other if they are observed in similar contexts with sufficient frequency. Our approach reparameterizes word embeddings using the same RNN machinery used in the parser: a word’s vector is calculated based on the sequence of orthographic symbols representing it (§3). Although our model is provided no supervision in the form of explicit morphological annotation, we find that it gives a large performance increase when parsing morphologically rich languages in the SPMRL datasets (Seddah et al., …

Content truncated to 500 words