Dependency-Based Relative Positional Encoding for Transformer NMT

Proceedings of Recent Advances in Natural Language Processing, pages 854–861, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_099 854 Dependency-Based Relative Positional Encoding for Transformer NMT Yutaro Omote and Akihiro Tamura and Takashi Ninomiya Ehime University {omote@ai.cs, tamura@cs, ninomiya@cs}.ehime-u.ac.jp Abstract In this paper, we propose a novel model for Transformer neural machine transla- tion that incorporates syntactic distances between two source words into the rel- ative position representations of a self- attention mechanism. In particular, the proposed model encodes pair-wise relative depths on a source dependency tree, which are the differences between the depths of two source words, in the encoder’s self- attention. Experiments show that our pro- posed model achieved a 0.5 point gain in BLEU on the Asian Scientific Paper Ex- cerpt Corpus Japanese-to-English transla- tion task. 1 Introduction Machine translation (MT) has been actively stud- ied for many decades. In recent years, neural ma- chine translation (NMT) has become dominant. In particular, the Transformer model (Vaswani et al., 2017), which is based solely on attention mech- anisms, has advanced the state-of-the-art perfor- mance on various translation tasks and has become the focus of many MT researchers nowadays. Un- like recurrent neural network (RNN) based mod- els (Sutskever et al., 2014; Luong et al., 2015) or convolutional neural network (CNN) based mod- els (Gehring et al., 2017), the Transformer model attends to words in the same sentence, i.e., a source sentence or a target sentence, through the self-attention mechanisms in each encoder and de- coder. In addition, it encodes the positional in- formation of each word, such as the word order, as positional encoding (PE) so that recurrent and convolutional structures are excluded and training can be parallelized. Since NMT appeared, trans- lation performance has been improved by using the syntactic information, such as phrase struc- tures or dependency structures, of the source-side, target-side, or both (Ding and Palmer, 2005; Chen et al., 2017; Eriguchi et al., 2017; Wu et al., 2018). In semantic role labeling (SRL), though the task is not MT, Strubell et al. (2018) improved a Transformer-based model through the learning of self-attention weighting on the basis of syntactic information, i.e., dependency structures. Hence, it is expected that the performance of Transformer NMT will be improved by incorporating syntax in- formation. In this paper, we aim to improve Transformer NMT by using dependency structures. Some re- searchers have improved Transformer NMT by modifying self-attention. Shaw et al. (2018) used relative position information between two words encoded in self-attention in addition to the abso- lute position information of words. Inspired by Shaw et al. (2018), we propose a novel Transformer NMT model that incorporates the relationships between two words on source de- pendency structures into relative position repre- sentations in self-attention. In particular, the pro- posed model adds a vector that encodes relative positional relationships between words on source dependency structures to a word embedding vec- tor. It adds only dependency information to the word embedding vector; hence, there is no need to change the whole Transformer’s mechanism …

Content truncated to 500 words