Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3059–3069 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3059 Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation Qiu Ran∗, Yankai Lin∗†, Peng Li∗†, Jie Zhou Pattern Recognition Center, WeChat AI, Tencent Inc., China {soulcaptran,yankailin,patrickpli,withtomzhou}@tencent.com Abstract Non-autoregressive neural machine translation (NAT) predicts the entire target sequence si- multaneously and significantly accelerates in- ference process. However, NAT discards the dependency information in a sentence, and thus inevitably suffers from the multi-modality problem: the target tokens may be provided by different possible translations, often caus- ing token repetitions or missing. To allevi- ate this problem, we propose a novel semi- autoregressive model RecoverSAT in this work, which generates a translation as a se- quence of segments. The segments are gen- erated simultaneously while each segment is predicted token-by-token. By dynamically de- termining segment length and deleting repet- itive segments, RecoverSAT is capable of re- covering from repetitive and missing token er- rors. Experimental results on three widely- used benchmark datasets show that our pro- posed model achieves more than 4× speedup while maintaining comparable performance compared with the corresponding autoregres- sive model. 1 Introduction Although neural machine translation (NMT) has achieved state-of-the-art performance in recent years (Cho et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017), most NMT models still suf- fer from the slow decoding speed problem due to their autoregressive property: the generation of a target token depends on all the previously gener- ated target tokens, making the decoding process intrinsically nonparallelizable. Recently, non-autoregressive neural machine translation (NAT) models (Gu et al., 2018; Li et al., 2019; Wang et al., 2019; Guo et al., 2019a; Wei et al., 2019) have been investigated to mitigate the ∗indicates equal contribution †indicates corresponding author Src. es gibt heute viele Farmer mit diesem Ansatz Feasible there are lots of farmers doing this today Trans. there are a lot of farmers doing this today Trans. 1 there are lots of of farmers doing this today Trans. 2 there are a lot farmers doing this today Table 1: A multi-modality problem example: NAT models generate each target token independently such that they may correspond to different feasible transla- tions, which usually manifests as repetitive (Trans. 1) or missing (Trans. 2) tokens. slow decoding speed problem by generating all tar- get tokens independently in parallel, speeding up the decoding process significantly. Unfortunately, these models suffer from the multi-modality prob- lem (Gu et al., 2018), resulting in inferior transla- tion quality compared with autoregressive NMT. To be specific, a source sentence may have mul- tiple feasible translations, and each target token may be generated with respect to different fea- sible translations since NAT models discard the dependency among target tokens. This generally manifests as repetitive or missing tokens in the translations. Table 1 shows an example. The Ger- man phrase “viele Farmer” can be translated as either “lots of farmers” or “a lot of farmers”. In the first …

Content truncated to 500 words