Dependency-Based Self-Attention for Transformer NMT

Proceedings of Recent Advances in Natural Language Processing, pages 239–246, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_028 239 Dependency-Based Self-Attention for Transformer NMT Hiroyuki Deguchi, Akihiro Tamura, Takashi Ninomiya Ehime University {deguchi@ai., tamura@, ninomiya@}cs.ehime-u.ac.jp Abstract In this paper, we propose a new Trans- former neural machine translation (NMT) model that incorporates dependency re- lations into self-attention on both source and target sides, dependency-based self- attention. The dependency-based self- attention is trained to attend to the modi- fiee for each token under constraints based on the dependency relations, inspired by linguistically-informed self-attention (LISA). While LISA was originally de- signed for Transformer encoder for se- mantic role labeling, this paper extends LISA to Transformer NMT by masking fu- ture information on words in the decoder- side dependency-based self-attention. Ad- ditionally, our dependency-based self- attention operates at subword units cre- ated by byte pair encoding. Experiments demonstrate that our model achieved a 1.0 point gain in BLEU over the base- line model on the WAT’18 Asian Scientific Paper Excerpt Corpus Japanese-to-English translation task. 1 Introduction In the field of machine translation (MT), the Transformer model (Vaswani et al., 2017) has outperformed recurrent neural network (RNN)- based models (Sutskever et al., 2014) and con- volutional neural network (CNN)-based models (Gehring et al., 2017) on many translation tasks, and thus has garnered attention from MT re- searchers. The Transformer model computes the strength of a relationship between two words in a sentence by means of a self-attention mecha- nism, which has contributed to the performance improvement in not only MT but also various NLP tasks such as language modeling and semantic role labeling (SRL). The performance of MT, including statisti- cal machine translation and RNN-based neural machine translation (NMT), has been improved by incorporating sentence structures (Lin, 2004; Chen et al., 2017; Eriguchi et al., 2017; Wu et al., 2018). In addition, Strubell et al. (2018) have im- proved a Transformer-based SRL model by in- corporating dependency structures of sentences into self-attention, which is called linguistically- informed self-attention (LISA). In LISA, one at- tention head of a multi-head self-attention is trained with constraints based on dependency rela- tions to attend to syntactic parents for each token. In the present work, we aim to improve trans- lation performance by utilizing dependency rela- tions in Transformer NMT. To this end, we pro- pose a Transformer NMT model that incorporates dependency relations into self-attention on both source and target sides. Specifically, in training, a part of self-attention is learned with constraints based on dependency relations of source or target sentences to attend to a modifiee for each token, and, in decoding, the proposed model translates a sentence in consideration of dependency relations in both the source and target sides, which are cap- tured by our self-attention mechanisms. Hereafter, the proposed self-attention is called dependency- based self-attention. Note that the dependency- based self-attention is inspired by LISA, but the straightforward adaptation of LISA, which is pro- posed for Transformer encoder, does not work well for NMT because a target sentence …

Content truncated to 500 words