Stack-based Multi-layer Attention for Transition-based Dependency Parsing

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1677–1682 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Stack-based Multi-layer Attention for Transition-based Dependency Parsing Zhirui Zhang1, Shujie Liu2, Mu Li2, Ming Zhou2, Enhong Chen1 University of Science and Technology of China, Hefei, China1 Microsoft Research Asia, Beijing, China2 zr011036@mail.ustc.edu.cn cheneh@ustc.edu.cn1 {shujie,muli,mingzhou}@microsoft.com2 Abstract Although sequence-to-sequence (seq2seq) network has achieved significant success in many NLP tasks such as machine trans- lation and text summarization, simply ap- plying this approach to transition-based dependency parsing cannot yield a compa- rable performance gain as in other state- of-the-art methods, such as stack-LSTM and head selection. In this paper, we propose a stack-based multi-layer atten- tion model for seq2seq learning to bet- ter leverage structural linguistics informa- tion. In our method, two binary vectors are used to track the decoding stack in transition-based parsing, and multi-layer attention is introduced to capture multiple word dependencies in partial trees. We conduct experiments on PTB and CTB datasets, and the results show that our pro- posed model achieves state-of-the-art ac- curacy and significant improvement in la- beled precision with respect to the baseline seq2seq model. 1 Introduction Deep learning models have been proven very ef- fective in solving various NLP problems such as language modeling, machine translation and syn- tactic parsing. For dependency parsing, one line of research aims to incrementally integrate dis- tributed word representations into classic depen- dency parsing (Chen and Manning, 2014; Weiss et al., 2015; Andor et al., 2016; Cross and Huang, 2016; Kiperwasser and Goldberg, 2016; Dozat and Manning, 2016). Another line of research at- tempts to leverage end-to-end neural network to perform dependency parsing, such as stack-LSTM and sequence-to-sequence (seq2seq) model (Dyer et al., 2015; Zhang et al., 2017; Wiseman and Rush, 2016). Recently seq2seq model has made significant success in many NLP tasks, such as machine translation and text summarization (Cho et al., 2014; Sutskever et al., 2014; Rush et al., 2015). Unfortunately, to our best knowledge, sim- ply applying seq2seq model to transition-based dependency parsing cannot achieve comparable results as in other state-of-the-art methods like stack-LSTM and head selection (Dyer et al., 2015; Zhang et al., 2017). One issue with the simple seq2seq neural net- work for dependency parsing is that structural lin- guistic information, which plays a key role in clas- sic transition-based or graph-based dependency parsing models, cannot be explicitly employed. For example, classic transition-based parsing al- gorithm utilizes a stack to manage the heads of partial sub-trees and leverages these evidents for action selection, while such information is missing from current seq2seq models. Another problem is related to the limit of the conventional attention network being used in seq2seq network, which is unable to capture dependencies between words in the input. As a matter of fact, various types of fea- tures (word unigram, bigram, trigram, ...) tradi- tionally adopted by transition-based parsing algo- rithm are usually ignored by the current attention mechanism, but they are very important to capture word dependencies …

Content truncated to 500 words