Improving Non-Autoregressive Neural Machine Translation via Modeling Localness

Proceedings of the 29th International Conference on Computational Linguistics, pages 5217–5226 October 12–17, 2022. 5217 Improving Non-Autoregressive Neural Machine Translation via Modeling Localness Yong Wang†∗ Xinwei Geng †Tencent Corporation, China {seaywang,xwgeng2014}@gmail.com Abstract Non-autoregressive translation (NAT) models, which eliminate the sequential dependencies within the target sentence, have achieved re- markable inference speed, but suffer from in- ferior translation quality. Towards exploring the underlying causes, we carry out a thor- ough preliminary study on the attention mech- anism, which demonstrates the serious weak- ness in capturing localness compared with con- ventional autoregressive translation (AT). In re- sponse to this problem, we propose to improve the localness of NAT models by explicitly in- troducing the information about surrounding words. Specifically, temporal convolutions are incorporated into both encoder and de- coder sides to obtain localness-aware represen- tations. Extensive experiments on several typi- cal translation datasets show that the proposed method can achieve consistent and significant improvements over strong NAT baselines. Fur- ther analyses on the WMT14 En⇒De transla- tion task reveal that compared with baselines, our approach accelerates the convergence in training and can achieve equivalent perfor- mance with a reduction of 70% training steps. 1 Introduction Based on the encoder-decoder framework (Cho et al., 2014; Sutskever et al., 2014), neural machine translation (NMT) (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) has achieved tremendous success in the past several years due to its excellent performance. Currently, state-of- the-art NMT systems are built in an autoregressive manner, which generates target tokens one by one from continuous representations summarized by the encoder. However, with the constraint of output tokens conditioned on previously generated ones, autoregressive translation (AT) inevitably suffers from serious latency during decoding, which be- comes a bottleneck of inference speed. ∗ Work done while at Microsoft Corporation. Towards accelerating the inference process, non-autoregressive neural machine translation (NAT) (Gu et al., 2018; Ghazvininejad et al., 2019; Du et al., 2021; Huang et al., 2022) has been pro- posed to break the above bottleneck. Instead of sequential generation in autoregressive translation, NAT models output the entire target sentence at once. Unfortunately, removing sequential depen- dencies within the target sentence brings NAT mod- els with serious weakness in capturing the highly multimodal distribution of target sentences (Gu et al., 2018). Accordingly, NAT models suffer from two kinds of incoherent translations, includ- ing repetitive translations and incomplete transla- tions (Wang et al., 2019), which leads to inferior translation performance. Since both types of translation errors do not com- monly appear in AT models, several works (Li et al., 2019; Wei et al., 2019) have proposed to leverage a well-trained AT model to enhance the training of NAT models. Inspired by this consideration, we empirically carry out a thorough study to present the weakness of NAT by investigating the distinc- tion between NAT and AT models. Specifically, we inspect the attention mechanism on two trans- lation tasks and reveal that, in contrast to autore- gressive models, NAT models lack the ability of either modeling …

Content truncated to 500 words