Bi-directional Attention with Agreement for Dependency Parsing

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2204–2214, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Bi-directional Attention with Agreement for Dependency Parsing Hao Cheng Hao Fang University of Washington {chenghao,hfang}@uw.edu Xiaodong He Jianfeng Gao Li Deng Microsoft Research {xiaohe,jfgao,deng}@microsoft.com Abstract We develop a novel bi-directional attention model for dependency parsing, which learns to agree on headword predictions from the for- ward and backward parsing directions. The parsing procedure for each direction is for- mulated as sequentially querying the memory component that stores continuous headword embeddings. The proposed parser makes use of soft headword embeddings, allowing the model to implicitly capture high-order pars- ing history without dramatically increasing the computational complexity. We conduct experiments on English, Chinese, and 12 other languages from the CoNLL 2006 shared task, showing that the proposed model achieves state-of-the-art unlabeled attachment scores on 6 languages.1 1 Introduction Recently, several neural network models have been developed for efficiently accessing long-term mem- ory and discovering dependencies in sequential data. The memory network framework has been studied in the context of question answering and language modeling (Weston et al., 2015; Sukhbaatar et al., 2015), whereas the neural attention model under the encoder-decoder framework has been applied to machine translation (Bahdanau et al., 2015) and constituency parsing (Vinyals et al., 2015b). Both frameworks learn the latent alignment between the source and target sequences, and the mechanism of 1Our software and models are available at https:// github.com/hao-cheng/biattdp. attention over the encoder can be viewed as a soft operation on the memory. Although already used in the encoder for capturing global context informa- tion (Bahdanau et al., 2015), the bi-directional recur- rent neural network (RNN) has yet to be employed in the decoder. Bi-directional decoding is expected to be advantageous over the previously developed uni-directional counterpart, because the former ex- ploits richer contextual information. Intuitively, we can use two separate uni-directional RNNs where each one constructs its respective attended encoder context vectors for computing RNN hidden states. However, the drawback of this approach is that the decoder would often produce different alignments resulting in discrepancies for the forward and back- ward directions. In this paper, we design a training objective function to enforce attention agreement between both directions, inspired by the alignment- by-agreement idea from Liang et al. (2006). Specif- ically, we develop a dependency parser (BiAtt-DP) using a bi-directional attention model based on the memory network. Given that the golden alignment is observed for dependency parsing in the training stage, we further derive a simple and interpretable approximation for the agreement objective, which makes a natural connection between the latent and observed alignment cases. The proposed BiAtt-DP parses a sentence in a linear order via sequentially querying the memory component that stores continuous embeddings for all headwords. In other words, we consider all pos- sible arcs during the parsing. This formulation is adopted by graph-based parsers such as the MST- Parser (McDonald et al., 2005). The consideration 2204 of all possible arcs …

Content truncated to 500 words