Hint-Based Training for Non-Autoregressive Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5708–5713, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5708 Hint-Based Training for Non-Autoregressive Machine Translation Zhuohan Li1 Zi Lin2,3 Di He2 Fei Tian4 Tao Qin5 Liwei Wang2 Tie-Yan Liu5 1UC Berkeley 2Peking University 3Google AI 4Facebook 5Microsoft Research zhuohan@cs.berkeley.edu {zi.lin, di he, wanglw}@pku.edu.cn feitia@fb.com {taoqin, tyliu}@microsoft.com Abstract Due to the unparallelizable nature of the autoregressive factorization, AutoRegressive Translation (ART) models have to generate tokens sequentially during decoding and thus suffer from high inference latency. Non- AutoRegressive Translation (NART) models were proposed to reduce the inference time, but could only achieve inferior translation ac- curacy. In this paper, we proposed a novel approach to leveraging the hints from hidden states and word alignments to help the training of NART models. The results achieve signif- icant improvement over previous NART mod- els for the WMT14 En-De and De-En datasets and are even comparable to a strong LSTM- based ART baseline but one order of magni- tude faster in inference. 1 Introduction Neural machine translation has attracted much at- tention in recent years (Bahdanau et al., 2014, 2016; Kalchbrenner et al., 2016; Gehring et al., 2016). Given a sentence x = (x1, . . . , xTx) from the source language, the straight-forward way for translation is to generate the target words y = (y1, . . . , yTy) one by one from left to right. This is also known as the AutoRegressive Translation (ART) models, in which the joint probability is de- composed into a chain of conditional probabilities: P(y|x) = ΠTy t=1P(yt|y<t, x), (1) While the ART models have achieved great suc- cess in terms of translation quality, the time con- sumption during inference is still far away from satisfactory. During training, the predictions at different positions can be estimated in parallel since the ground truth pair (x, y) is exposed to the model. However, during inference, the model The work was performed at Microsoft Research Asia. has to generate tokens sequentially as y<t must be inferred on the fly. Such autoregressive behavior becomes the bottleneck of the computational time (Wu et al., 2016). In order to speed up the inference process, a line of works begin to develop non-autoregressive translation models. These models break the au- toregressive dependency by decomposing the joint probability with P(y|x) = P(Ty|x)ΠTy t=1P(yt|x). (2) The lost of autoregressive dependency largely hurt the consistency of the output sentences, in- crease the difficulty in the learning process and thus lead to a low quality translation. Previous works mainly focus on adding different compo- nents into the NART model to improve the ex- pressiveness of the network structure to overcome the loss of autoregressive dependency (Gu et al., 2017; Lee et al., 2018; Kaiser et al., 2018). How- ever, the computational overhead of new compo- nents will hurt the inference speed, contradicting with the goal of the NART models: to …

Content truncated to 500 words