Smart-Start Decoding for Neural Machine Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3982–3988 June 6–11, 2021. ©2021 Association for Computational Linguistics 3982 Smart-Start Decoding for Neural Machine Translation Jian Yang1 ∗, Shuming Ma2, Dongdong Zhang2, Juncheng Wan3, Zhoujun Li1 †, Ming Zhou2 1State Key Lab of Software Development Environment, Beihang University 2Microsoft Research Asia 3Shanghai Jiao Tong University {jiaya, lizj}@buaa.edu.cn; {shumma, dozhang, mingzhou}@microsoft.com; junchengwan@apex.sjtu.edu.cn Abstract Most current neural machine translation mod- els adopt a monotonic decoding order of either left-to-right or right-to-left. In this work, we propose a novel method that breaks up the lim- itation of these decoding orders, called Smart- Start decoding. More specifically, our method first predicts a median word. It starts to decode the words on the right side of the median word and then generates words on the left. We evalu- ate the proposed Smart-Start decoding method on three datasets. Experimental results show that the proposed method can significantly out- perform strong baseline models. 1 Introduction Neural machine translation (NMT) has made re- markable progress in recent years. There has been much progress in encoder-decoder framework, in- cluding recurrent neural models (Wu et al., 2016), convolutional models (Gehring et al., 2017) and self-attention models (Vaswani et al., 2017). Par- ticularly, the Transformer, only relying on self- attention networks, has achieved state-of-the-art performance on different benchmarks. Most encoder-decoder frameworks generate tar- get translation in a completely monotonic order from left to right (L2R) or from right to left (R2L). However, monotonic generation is not always the best translation order for the machine translation task. As shown in Figure 1, “乐 (happy)” needs to leverage the future context “开朗 (lively)” to make disambiguation of the translation in English sentence, because “乐” has two meanings: “happy to do something” and “Le (person name)”. In this example, the L2R baseline model produced an in- correct translation of “Le (person name)” due to unseen future context. Asia.∗Contribution during internship at Microsoft Research †Corresponding author. Source: � �� �� , �� �� �� �� . Ref: Happy to talk with people , Yang Sen has a lively personality Left-to-Right Translation: Le talks with people , Yang Sen is very lively . Translation: Chatting with people , Yang Sen has a lively personality . glad with people chat Yang Sen personality very ������ (a) (b) Smart-Start: Yang Sen has a lively personality . [m] Chatting with people , Figure 1: Example of baseline method (a) and our Smart-Start method (b). “[m]” is designed to indicate the termination of the right part generation. “[m]” is an abbreviation of “[middle]”. There are some related works on non-monotonic text generation (Mehri and Sigal, 2018; Welleck et al., 2019; Gu et al., 2019; Zhou et al., 2019b,a). Inspired by these works, we are extremely inter- ested in considering choosing one proper position to start decoding instead of L2R or R2L order. We propose a novel method called the Smart-Start de- coding method. Specifically, our method starts the generation …

Content truncated to 500 words