Learning to Rewrite for Non-Autoregressive Neural Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3297–3308 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3297 Learning to Rewrite for Non-Autoregressive Neural Machine Translation Xinwei Geng† Xiaocheng Feng†‡ Bing Qin†‡ † Harbin Institute of Technology ‡ Peng Cheng Laboratory {xwgeng,xcfeng,qinb}@ir.hit.edu.cn Abstract Non-autoregressive neural machine transla- tion, which decomposes the dependence on previous target tokens from the inputs of the decoder, has achieved impressive inference speedup but at the cost of inferior accuracy. Previous works employ iterative decoding to improve the translation by applying multiple refinement iterations. However, a serious drawback is that these approaches expose the serious weakness in recognizing the erroneous translation pieces. In this paper, we propose an architecture named REWRITENAT to explic- itly learn to rewrite the erroneous translation pieces. Specifically, REWRITENAT utilizes a locator module to locate the erroneous ones, which are then revised into the correct ones by a revisor module. Towards keeping the consis- tency of data distribution with iterative decod- ing, an iterative training strategy is employed to further improve the capacity of rewriting. Extensive experiments conducted on several widely-used benchmarks show that REWRITE- NAT can achieve better performance while sig- nificantly reducing decoding time, compared with previous iterative decoding strategies. In particular, REWRITENAT can obtain compet- itive results with autoregressive translation on WMT14 En↔De, En→Fr and WMT16 Ro→En translation benchmarks1. 1 Introduction State-of-the-art neural machine translation (NMT) systems use autoregressive decoding where the de- coder generates a target sentence word by word, and the generation of the latter words depends on previously generated ones (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). In- stead of sequential decoding as in the autoregres- sive translation (AT), non-autoregressive neural ma- chine translation (NAT) (Gu et al., 2018; Guo et al., 2019; Ma et al., 2019; Wei et al., 2019; Sun et al., 1Our code is publicly available at https://github. com/xwgeng/RewriteNAT. Encoder x1 x2 x3 x4 Revisor y1 y4 y5 encoder-decoder attention ̂y2 ̂y3 y1 y4 y5 [MASK] [MASK] Locator Encoder x1 x2 x3 x4 Decoder y5 encoder-decoder attention ̂y2 ̂y3 y1 y4 y5 [MASK] Heuristic Rules ̂y3[MASK] ̂y2 translation translation (a) Masked LM-based NAT Encoder x1 x2 x3 x4 Revisor y1 y4 y5 encoder-decoder attention ̂y2 ̂y3 y1 y4 y5 [MASK] [MASK] Locator Encoder x1 x2 x3 x4 Decoder y5 encoder-decoder attention ̂y2 ̂y3 y1 y4 y5 [MASK] Heuristic Rules ̂y3[MASK] ̂y2 translation translation (b) RewriteNAT Figure 1: Illustration of the difference in masking words between (a) conventional masked LM-based NAT (Ghazvininejad et al., 2019) and (b) our proposed REWRITENAT. Instead of using inefficient heuristic rules which perhaps mask correct words in some case (e.g., y1y4), REWRITENAT utilizes an additional loca- tor module to learn to explicitly distinguish erroneous translation pieces (e.g., ˆy2ˆy3), annotated as special sym- bol (i.e., [MASK]). 2019; Ghazvininejad et al., 2020a; Zhou et al., 2020; Ding et al., 2021a,b) generates the whole target sentence simultaneously. To enable paral- lel decoding, NAT imposes a conditional indepen- dence assumption among words in …

Content truncated to 500 words