Incorporating Source Syntax into Transformer-Based Neural Machine Translation

Proceedings of the Fourth Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 24–33 Florence, Italy, August 1-2, 2019. c⃝2019 Association for Computational Linguistics 24 Incorporating Source Syntax into Transformer-Based Neural Machine Translation Anna Currey University of Edinburgh a.currey@sms.ed.ac.uk Kenneth Heafield University of Edinburgh kheafiel@ed.ac.uk Abstract Transformer-based neural machine transla- tion (NMT) has recently achieved state-of- the-art performance on many machine trans- lation tasks. However, recent work (Ra- ganato and Tiedemann, 2018; Tang et al., 2018; Tran et al., 2018) has indicated that Transformer models may not learn syntac- tic structures as well as their recurrent neu- ral network-based counterparts, particularly in low-resource cases. In this paper, we incor- porate constituency parse information into a Transformer NMT model. We leverage lin- earized parses of the source training sentences in order to inject syntax into the Transformer architecture without modifying it. We introduce two methods: a multi-task ma- chine translation and parsing model with a sin- gle encoder and decoder, and a mixed encoder model that learns to translate directly from parsed and unparsed source sentences. We evaluate our methods on low-resource trans- lation from English into twenty target lan- guages, showing consistent improvements of 1.3 BLEU on average across diverse target lan- guages for the multi-task technique. We fur- ther evaluate the models on full-scale WMT tasks, finding that the multi-task model aids low- and medium-resource NMT but degener- ates high-resource English→German transla- tion. 1 Introduction Transformer-based neural machine translation (NMT) (Vaswani et al., 2017) has recently out- performed recurrent neural network (RNN)-based models (Bahdanau et al., 2015; Cho et al., 2014) in many tasks (Bojar et al., 2018). However, there is still room for improvement for NMT, partic- ularly for low- and moderate-resource language pairs. Enriching NMT with syntactic informa- tion has the potential to improve generalization in low-resource scenarios, and adding syntax to Transformer-based NMT is currently an underex- plored research area. Transformer-based NMT may in fact stand to benefit even more from explicit syntactic annota- tions than RNN-based NMT, particularly in low- resource settings. On the one hand, the Trans- former model already learns some syntax with- out explicit supervision in high-resource cases. Vaswani et al. (2017) visualized a few encoder self-attentions in a trained NMT model and found that they seemed to capture syntactic structure. This was formalized by Raganato and Tiedemann (2018), who found that Transformer encoders trained on high-resource NMT tasks were able to perform reasonably well at part-of-speech tagging, chunking, and other tasks. However, for Trans- formers trained on low-resource NMT, the results on these tasks were not as strong. Additionally, Tran et al. (2018) found that an RNN language model did better at predicting subject-verb agree- ment than a Transformer language model; Tang et al. (2018) saw similar results for Transformer vs. RNN NMT models. Thus, the goal of this paper is to improve Transformer-based NMT using source-side syn- tactic supervision. We propose two methods that incorporate source-side linearized constituency parses into Transformer-based NMT. The first, multi-task, uses the Transformer to …

Content truncated to 500 words