Imitation Learning for Non-Autoregressive Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1304–1312 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1304 Imitation Learning for Non-Autoregressive Neural Machine Translation Bingzhen Wei1, Mingxuan Wang, Hao Zhou, Junyang Lin1,3, Xu Sun1,2 1MOE Key Lab of Computational Linguistics, School of EECS, Peking University 2Deep Learning Lab, Beijing Institute of Big Data Research, Peking University 3School of Foreign Languages, Peking University {weibz,linjunyang,xusun}@pku.edu.cn xuanxuans27@gmail.com, haozhou0806@gmail.com Abstract Non-autoregressive translation models (NAT) have achieved impressive inference speedup. A potential issue of the existing NAT algo- rithms, however, is that the decoding is con- ducted in parallel, without directly consider- ing previous context. In this paper, we pro- pose an imitation learning framework for non- autoregressive machine translation, which still enjoys the fast translation speed but gives comparable translation performance compared to its auto-regressive counterpart. We con- duct experiments on the IWSLT16, WMT14 and WMT16 datasets. Our proposed model achieves a significant speedup over the autore- gressive models, while keeping the translation quality comparable to the autoregressive mod- els. By sampling sentence length in parallel at inference time, we achieve the performance of 31.85 BLEU on WMT16 Ro→En and 30.68 BLEU on IWSLT16 En→De. 1 Introduction Neural machine translation (NMT) with encoder- decoder architectures (Sutskever et al., 2014; Cho et al., 2014) achieve significantly improved perfor- mance compared with traditional statistical meth- ods(Koehn et al., 2003; Koehn, 2010). Never- theless, the autoregressive property of the NMT decoder has been a bottleneck of the translation speed. Specifically, the decoder, whether based on Recurrent Neural Network (RNN) (Hochreiter and Schmidhuber, 1997; Cho et al., 2014) or attention mechanism (Vaswani et al., 2017), sequentially generates words. The latter words are conditioned on previous words in a sentence. Such bottleneck disables parallel computation of decoder, which is serious for NMT, since the NMT decoding with a large vocabulary is extremely time-consuming. Recently, a line of research work (Gu et al., 2017; Lee et al., 2018; Libovick and Helcl, 2018; (a) Autoregressive NMT (b) Non-Autoregressive NMT Figure 1: Neural architectures for Autoregressive NMT and Non-Autoregressive NMT. Wang et al., 2018) propose to break the autoregres- sive bottleneck by introducing non-autoregressive neural machine translation (NAT). In NAT, the de- coder generates all words simultaneously instead of sequentially. Intuitively, NAT abandon feed- ing previous predicted words into decoder state at the next time step, but directly copy source en- coded representation (Gu et al., 2017; Lee et al., 2018; Guo et al., 2018; Wang et al., 2019) as inputs of the decoder. Thus, the generation of the NAT models does not condition on previous prediction. NAT enables parallel computation of decoder, giv- ing significantly fast translation speed with mod- erate accuracy (always within 5 BLEU). Figure 1 shows the difference between autoregressive and non-autoregressive models. However, we argue that current NAT ap- proaches suffer from delayed supervisions (or re- wards) and large search space in training. NAT decoder simultaneously generates all words of the translation, the search space of …

Content truncated to 500 words