Progressive Multi-Granularity Training for Non-Autoregressive Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2797–2803 August 1–6, 2021. ©2021 Association for Computational Linguistics 2797 Progressive Multi-Granularity Training for Non-Autoregressive Translation Liang Ding∗ The University of Sydney ldin3097@sydnye.edu.au Longyue Wang∗ Tencent AI Lab vinnylywang@tencent.com Xuebo Liu University of Macau nlp2ct.xuebo@gmail.com Derek F. Wong University of Macau derekfw@um.edu.com Dacheng Tao JD Explore Academy, JD.com dacheng.tao@gmail.com Zhaopeng Tu Tencent AI Lab zptu@tencent.com Abstract Non-autoregressive translation (NAT) signif- icantly accelerates the inference process via predicting the entire target sequence. How- ever, recent studies show that NAT is weak at learning high-mode of knowledge such as one-to-many translations. We argue that modes can be divided into various granu- larities which can be learned from easy to hard. In this study, we empirically show that NAT models are prone to learn fine-grained lower-mode knowledge, such as words and phrases, compared with sentences. Based on this observation, we propose progressive multi- granularity training for NAT. More specifi- cally, to make the most of the training data, we break down the sentence-level examples into three types, i.e. words, phrases, sen- tences, and with the training goes, we pro- gressively increase the granularities. Experi- ments on Romanian-English, English-German, Chinese-English and Japanese-English demon- strate that our approach improves the phrase translation accuracy and model reordering abil- ity, therefore resulting in better translation quality against strong NAT baselines. Also, we show that more deterministic fine-grained knowledge can further enhance performance. 1 Introduction Non-autoregressive translation (NAT, Gu et al., 2018) has been proposed to improve the decoding efficiency by predicting all tokens independently and simultaneously. Different from autoregressive translation (AT, Vaswani et al., 2017) models that generate each target word conditioned on previ- ously generated ones, NAT models suffer from the multimodality problem (i.e. multiple transla- tions for a single input), in which the conditional ∗ Liang Ding and Longyue Wang contributed equally to this work. Work was done when Liang Ding and Xuebo Liu were interning at Tencent AI Lab. Granular. AT NAT Raw △ KD △ WORD 59.8 57.1 -2.7 59.0 -0.8 PHRASE 36.0 31.7 -4.3 34.2 -1.8 SENTENCE 29.2 24.5 -4.7 27.0 -2.2 Table 1: Translation performance at different granular- ity on the WMT14 English⇒German dataset. “△” in- dicates the performance gap between the NAT and AT. independence assumption prevents a model from properly capturing the highly multimodal distribu- tion of target translations. To reduce the modes of training data, sequence-level knowledge distillation (KD) (Kim and Rush, 2016) is widely employed via replacing their original target samples with sen- tences generated from an AT teacher (Gu et al., 2018; Zhou et al., 2020; Ren et al., 2020). Although KD reduces the learning difficulty for NAT, there are still complicated word orders and structures (Gell-Mann and Ruhlen, 2011) in the synthetic sentences, making the NAT performance sub-optimal. To answer this challenge, Saharia et al. (2020); Ran et al. (2021) propose to lowers the bilingual modeling difficulties under the mono- tonicity assumption, where bilingual sentences are in the same word order. However, they make ex- …

Content truncated to 500 words