Synchronous Refinement for Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL 2022, pages 2986 - 2996 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Synchronous Refinement for Neural Machine Translation Kehai Chen1, Masao Utiyama2, and Eiichiro Sumita2, Rui Wang3, and Min Zhang1∗ 1Harbin Institute of Technology, Shenzhen, China 2National Institute of Information and Communications Technology, Kyoto, Japan 3Shanghai Jiao Tong University, Shanghai, China {chenkehai,zhangmin2021}@hit.edu.cn {mutiyama, eiichiro.sumita}@nict.go.jp wangrui12@sjtu.edu.cn Abstract Machine translation typically adopts an encoder-to-decoder framework, in which the decoder generates the target sentence word-by-word in an auto-regressive manner. However, the auto-regressive decoder faces a deep-rooted one-pass issue whereby each generated word is considered as one element of the final output regardless of whether it is correct or not. These generated wrong words further constitute the target historical context to affect the generation of subsequent target words. This paper proposes a novel synchronous refinement method to revise potential errors in the generated words by considering part of the target future context. Particularly, the proposed approach allows the auto-regressive decoder to refine the previously generated target words and generate the next target word synchronously. The experimental results on three widely-used machine translation tasks demonstrated the effectiveness of the proposed approach. 1 Introduction Recently, the encoder-decoder framework has obtained impressive results over various machine translation tasks (Barrault et al., 2020; Akhbardeh et al., 2021). Typically, decoder first represents those generated target words as a dependent-time target representation, and then uses an attention mechanism to summarize a dependent-time context from the source input for generating the next target word. Since this generated target word is conditioned on previously generated target words at each time step, the decoding process is often called auto-regressive decoding. Finally, decoder generates a target language sentence word-by-word in the auto-regressive decoding manner (Bahdanau et al., 2015; Vaswani et al., 2017). However, the auto-regressive decoder often encounters an inherent one-pass issue whereby ∗Corresponding author each generated target word is one element of the final output of the machine translation model regardless of whether it is correct or not. These generated wrong target words are further added to the target historical context to affect the generation of subsequent target words, which hinders the performance of machine translation. Take a Chinese-to-English translation case in Figure 1 generated by a trained neural machine translation (NMT) model (Vaswani et al., 2017), we illustrate the once-generation issue. In the generated target translation “Tgt", there is first an inappropriate translation “clean up" compared with “monitor" in the reference “Ref". The “clean up" is regarded as the final translation to confuse the understanding of the meaning of the source sentence. When these inappropriate or incorrect target words constitute part of the target historical context, the once- generation issue further affects the generation of subsequent target words, for example, “drivers’ mobile phone" is far away from the meaning of source sentence “the driver plays mobile phone while driving". To verify this issue, we artificially revised the inappropriate translation “clean up" as “monitor" during the decoding, and observed that the subsequent translation “driver …

Content truncated to 500 words