Glancing Transformer for Non-Autoregressive Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1993–2003 August 1–6, 2021. ©2021 Association for Computational Linguistics 1993 Glancing Transformer for Non-Autoregressive Neural Machine Translation Lihua Qian1,2∗ Hao Zhou2 Yu Bao3 Mingxuan Wang2 Lin Qiu1 Weinan Zhang1 Yong Yu1 Lei Li2 1 Shanghai Jiao Tong University 2 ByteDance AI Lab 3 Nanjing University {qianlihua,lqiu,wnzhang,yyu}@apex.sjtu.edu.cn {zhouhao.nlp,wangmingxuan.89,lileilab}@bytedance.com baoy@smail.nju.edu.cn Abstract Recent work on non-autoregressive neural ma- chine translation (NAT) aims at improving the efficiency by parallel decoding without sac- rificing the quality. However, existing NAT methods are either inferior to Transformer or require multiple decoding passes, leading to reduced speedup. We propose the Glancing Language Model (GLM) for single-pass par- allel generation models. With GLM, we de- velop Glancing Transformer (GLAT) for ma- chine translation. With only single-pass par- allel decoding, GLAT is able to generate high-quality translation with 8×-15× speedup. Note that GLAT does not modify the net- work architecture, which is a training method to learn word interdependency. Experiments on multiple WMT language directions show that GLAT outperforms all previous single pass non-autoregressive methods, and is nearly comparable to Transformer, reducing the gap to 0.25-0.9 BLEU points. 1 Introduction Transformer has been the most widely used ar- chitecture for machine translation (Vaswani et al., 2017). Despite its strong performance, the decod- ing of Transformer is inefficient as it adopts the sequential auto-regressive factorization for its prob- ability model (Figure 1a). Recent work such as the non-autoregressive transformer (NAT), aims to decode target tokens in parallel to speed up the gen- eration (Gu et al., 2018). However, the vanilla NAT still lags behind the Transformer in translation qual- ity – with a gap of about 7.0 BLEU points. NAT assumes the conditional independence of the target tokens given the source sentence. We suspect that NAT’s conditional independence assumption pre- vents learning word interdependency in the target ∗The work was done when the first author was an intern at Bytedance. ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 h1 h2 h3 h4 h5 NAT Decoding H H′� Glancing Sampling Hamming Distance N( ̂Y, Y) = 3 y1 y2 y3 y4 y5 ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 y1 y3 y5 Replace Inputs 0.8 0.5 0.7 0.6 0.9 h2 h4 y1 y2 y1 y2 y5 Decoder Encoder x1 x2 x3 x4 y3 y4 y3 y4 [BOS] y1 y2 y5 Decoder Encoder x1 x2 x3 x4 y3 y4 h2 h4 h1 h3 h5 attention Random Masking Decoder Encoder x1 x2 x3 x4 y1 y4 y5 [MASK][MASK] y2 y3 y1 y5 y4 attention an apple in the car ein Apfel im Auto Decoder Encoder x1 x2 x3 x4 y1 h2 y5 h4 y3 y2 y4 Glancing Sampling y3 y1 y5 ein Apfel im Auto an apple in the car attention ein Apfel im Auto ein Apfel im Auto apple in apple the an the car an apple in the an in car attention (a) Sequential LM ̂y1 ̂y2 ̂y4 ̂y5 ̂y3 h1 h2 h3 h4 …

Content truncated to 500 words