Transition-based Parsing with Stack-Transformers

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1001–1007 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1001 Transition-based Parsing with Stack-Transformers Ram´on Fernandez Astudillo† Miguel Ballesteros∗♦ Tahira Naseem† Austin Blodget∗♥ Radu Florian† †IBM Research ♦Amazon AI ♥Georgetown University {ramon.astudillo,tnassem,raduf}@ibm.com Abstract Modeling the parser state is key to good perfor- mance in transition-based parsing. Recurrent Neural Networks considerably improved the performance of transition-based systems by modelling the global state, e.g. stack-LSTM parsers, or local state modeling of contextual- ized features, e.g. Bi-LSTM parsers. Given the success of Transformer architectures in re- cent parsing systems, this work explores mod- ifications of the sequence-to-sequence Trans- former architecture to model either global or local parser states in transition-based parsing. We show that modifications of the cross atten- tion mechanism of the Transformer consider- ably strengthen performance both on depen- dency and Abstract Meaning Representation (AMR) parsing tasks, particularly for smaller models or limited training data. 1 Introduction Transition-based Parsing transforms the task of pre- dicting a graph from a sentence into predicting an action sequence of a state machine that produces the graph (Nivre, 2003, 2004; Kubler et al., 2009; Henderson et al., 2013). These parsers are attrac- tive for their linear inference time and interpretabil- ity, however, their performance hinges on effective modeling of the parser state at every decision step. Parser states typically comprise two memories, a buffer and a stack, from which tokens can be pushed or popped (Kubler et al., 2009). Tradi- tionally, parser states were modeled using hand selected local features pertaining only to the words on the top of the stack or buffer (Nivre et al., 2007; Zhang and Nivre, 2011, inter-alia). With the widespread use of neural networks, global models of the parser state such as the stack-LSTM (Dyer et al., 2015) allowed encoding the entire buffer and ∗Miguel’s and Austin’s contributions were carried out while at IBM Research. stack. It was later shown that local features of the stack and buffer extracted from contextual word representations, such as Bi-LSTMs, could outper- form global modeling (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2016). With the rise of the Transformer model (Vaswani et al., 2017), various approaches have been pro- posed that leverage this architecture for parsing (Kondratyuk, 2019; Kulmizev et al., 2019; Mrini et al., 2019; Ahmad et al., 2019; Cai and Lam, 2020). In this work we revisit the local versus global paradigms of state modeling in the context of sequence-to-sequence Transformers applied to action prediction for transition-based parsing. Sim- ilarly to previous works for RNN sequence to se- quence (Liu and Zhang, 2017; Zhang et al., 2017), we propose a modification of the cross-attention mechanism of the Transformer to provide global parser state modeling. We analyze the role of lo- cal versus global parser state modeling, stack and buffer modeling, effects model size as well as task complexity and amount of training data. Results show that local and global state model- ing of the parser state yield more than 2 …

Content truncated to 500 words