Dynamic Feature Selection with Attention in Incremental Parsing

Proceedings of the 27th International Conference on Computational Linguistics, pages 785–794 Santa Fe, New Mexico, USA, August 20-26, 2018. 785 Dynamic Feature Selection with Attention in Incremental Parsing Ryosuke Kohita†, Hiroshi Noji§ and Yuji Matsumoto‡ †IBM Research §Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST) †‡Graduate School of Information Science, Nara Institute of Science and Technology (NAIST) ryosuke.kohita1@ibm.com, hiroshi.noji@aist.go.jp, matsu@is.naist.jp Abstract One main challenge for incremental transition-based parsers, when future inputs are invisible, is to extract good features from a limited local context. In this work, we present a simple technique to maximally utilize the local features with an attention mechanism, which works as context- dependent dynamic feature selection. Our model learns, for example, which tokens should a parser focus on, to decide the next action. Our multilingual experiment shows its effectiveness across many languages. We also present an experiment with augmented test dataset and demon- strate it helps to understand the model’s behavior on locally ambiguous points. 1 Introduction This paper explores better feature representations for incremental dependency parsing. We focus on a system that builds a parse tree incrementally receiving each word of a sentence, which is crucial for interactive systems to achieve fast response or human-like behavior such as understanding from partial input (Baumann, 2013). The most natural way to achieve incremental parsing is using a transition system (Nivre, 2008), and for such parsers, the main challenge is to choose an appropriate action with only the local context information. While some recent transition-based parsers alleviate this difficulty by exploiting the entire input sentence with recurrent neural networks (Kiperwasser and Goldberg, 2016; Shi et al., 2017), one possible disadvantage is to require that all inputs are visible from the beginning, which should be problem when we try more strict incremental conditions such as simultaneous translation. Therefore there are still demands to explore the effective way to extract better feature representation from incomplete inputs. In this paper, we incorporate a simple attention mechanism (Bahdanau et al., 2015) with an incremental parser and investigate its effectiveness during the feature extraction. Attention mechanism itself has firstly succeeded in machine translation, capturing relative importances of tokens on a certain step for a proper output (Bahdanau et al., 2015; Luong et al., 2015). The characteristic to weight on some features automatically and effectively can be applied to various tasks such as seq-to-seq parsing model (Vinyals et al., 2015), text summarization (Rush et al., 2015), dialogue generation (Shang et al., 2015), image captioning (Xu et al., 2015) in which the systems can enjoy performance gain by attending to specific clues depending on a given situation. We can also expect this behavior is helpful to fix the error which transition-based parsers often commits due to local ambiguities. John on Monday introduces advisors. 40nmod 40obl Figure 1: A locally ambiguous sentence. “Monday” should be analyzed as oblique of “introduce” while tends to be analyzed as a noun modifier of “John”. This work is licensed under a Creative Commons Attribution 4.0 International …

Content truncated to 500 words