Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3013–3024 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3013 Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation Chenze Shao123, Yang Feng12⋆, Jinchao Zhang3, Fandong Meng3, Xilin Chen12 and Jie Zhou3 1 University of Chinese Academy of Sciences 2 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 3 Pattern Recognition Center, WeChat AI, Tencent Inc, China {shaochenze18z, fengyang, xlchen}@ict.ac.cn {dayerzhang, fandongmeng, withtomzhou}@tencent.com Abstract Non-Autoregressive Transformer (NAT) aims to accelerate the Transformer model through discarding the autoregressive mechanism and generating target words independently, which fails to exploit the target sequential informa- tion. Over-translation and under-translation errors often occur for the above reason, espe- cially in the long sentence translation scenario. In this paper, we propose two approaches to retrieve the target sequential information for NAT to enhance its translation ability while preserving the fast-decoding property. Firstly, we propose a sequence-level training method based on a novel reinforcement algorithm for NAT (Reinforce-NAT) to reduce the variance and stabilize the training procedure. Sec- ondly, we propose an innovative Transformer decoder named FS-decoder to fuse the target sequential information into the top layer of the decoder. Experimental results on three translation tasks show that the Reinforce-NAT surpasses the baseline NAT system by a sig- nificant margin on BLEU without decelerat- ing the decoding speed and the FS-decoder achieves comparable translation performance to the autoregressive Transformer with consid- erable speedup. 1 Introduction Neural machine translation (NMT) models (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2014) solve the machine translation problem with the Encoder-Decoder framework and achieve im- pressive performance on translation quality. Re- cently, the Transformer model (Vaswani et al., 2017) further enhances the translation perfor- mance on multiple language pairs, while suffer- ing from the slow decoding procedure, which re- Joint work with Pattern Recognition Center, WeChat AI, Tencent Inc, China. ⋆ Corresponding Author Src und noch tragischer ist , dass es Oxford war · · · Ref even more tragic is that it was Oxford · · · NAT and more more more more that it was Oxford · · · AR and , more tragic , Oxford was · · · Table 1: A fragment of a long sentence translation. AR stands for the translation of the autoregressive Trans- former. The output of the NAT model contains re- peated translations of word ‘more’ and misses the word ‘tragic’. stricts its application scenarios. The slow decod- ing problem of the Transformer model is caused by its autoregressive nature, which means that the target sentence is generated word by word accord- ing to the source sentence representations and the target translation history. Non-autoregressive Transformer model (Gu et al., 2017a) is proposed to accelerate the de- coding process, which can simultaneously gen- erate target words by discarding the autoregres- sive mechanism. Since the generation of target words is independent, NAT models utilize alter- native information such …

Content truncated to 500 words