Enhancing Machine Translation with Dependency-Aware Self-Attention

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1618–1627 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1618 Enhancing Machine Translation with Dependency-Aware Self-Attention Emanuele Bugliarello∗ University of Copenhagen emanuele@di.ku.dk Naoaki Okazaki Tokyo Institute of Technology okazaki@c.titech.ac.jp Abstract Most neural machine translation models only rely on pairs of parallel sentences, as- suming syntactic information is automati- cally learned by an attention mechanism. In this work, we investigate different ap- proaches to incorporate syntactic knowledge in the Transformer model and also propose a novel, parameter-free, dependency-aware self-attention mechanism that improves its translation quality, especially for long sen- tences and in low-resource scenarios. We show the efficacy of each approach on WMT English↔German and English→Turkish, and WAT English→Japanese translation tasks. 1 Introduction Research in neural machine translation (NMT) has mostly exploited corpora consisting of pairs of par- allel sentences, with the assumption that a model can automatically learn prior linguistic knowledge via an attention mechanism (Luong et al., 2015). However, Shi et al. (2006) found that these models still fail to capture deep structural details, and sev- eral studies (Sennrich and Haddow, 2016; Eriguchi et al., 2017; Chen et al., 2017, 2018) have shown that syntactic information has the potential to im- prove these models. Nevertheless, the majority of syntax-aware NMT models are based on recurrent neural networks (RNNs; Elman 1990), with only a few recent studies that have investigated methods for the Transformer model (Vaswani et al., 2017). Wu et al. (2018) evaluated an approach to incor- porate syntax in NMT with a Transformer model, which not only required three encoders and two decoders, but also target-side dependency rela- tions (precluding its use to low-resource target lan- guages). Zhang et al. (2019) integrate source-side syntax by concatenating the intermediate represen- tations of a dependency parser to word embeddings. ∗Work done while at Tokyo Institute of Technology. In contrast to ours, this approach does not allow to learn sub-word units at the source side, requiring a larger vocabulary to minimize out-of-vocabulary words. Saunders et al. (2018) interleave words with syntax representations which results in longer sequences – requiring gradient accumulation for ef- fective training – while only leading to +0.5 BLEU on WAT Ja-En when using ensembles of Transform- ers. Finally, Currey and Heafield (2019) propose two simple data augmentation techniques to incor- porate source-side syntax: one that works well on low-resource data, and one that achieves a high score on a large-scale task. Our approach, on the other hand, performs equally well in both settings. While these studies improve the translation qual- ity of the Transformer, they do not exploit its prop- erties. In response, we propose to explicitly en- hance the its self-attention mechanism (a core com- ponent of this architecture) to include syntactic in- formation without compromising its flexibility. Re- cent studies have, in fact, shown that self-attention networks benefit from modeling local contexts by reducing the dispersion of the attention distribu- tion (Shaw et al., 2018; Yang et al., 2018, 2019), …

Content truncated to 500 words