Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4811 - 4823 December 7-11, 2022 ©2022 Association for Computational Linguistics Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translation Huanran Zheng1, Wei Zhu1, Pengfei Wang1, Xiaoling Wang1 1East China Normal University, Shanghai, China {hrzheng,wzhu,pfwang}@stu.ecnu.edu.cn {xlwang}@cs.ecnu.edu.cn Abstract Non-autoregressive translation (NAT) model achieves a much faster inference speed than the autoregressive translation (AT) model be- cause it can simultaneously predict all tokens during inference. However, its translation qual- ity suffers from degradation compared to AT. And existing NAT methods only focus on im- proving the NAT model’s performance but do not fully utilize it. In this paper, we propose a simple but effective method called “Candi- date Soups,” which can obtain high-quality translations while maintaining the inference speed of NAT models. Unlike previous ap- proaches that pick the individual result and dis- card the remainders, Candidate Soups (CDS) can fully use the valuable information in the different candidate translations through model uncertainty. Extensive experiments on two benchmarks (WMT’14 EN–DE and WMT’16 EN–RO) demonstrate the effectiveness and gen- erality of our proposed method, which can sig- nificantly improve the translation quality of var- ious base models. More notably, our best vari- ant outperforms the AT model on three transla- tion tasks with 7.6× speedup.1 1 Introduction Autoregressive translation (AT) models based on Transformer (Vaswani et al., 2017; So et al., 2019; Sun et al., 2022; Zhu et al., 2021a), where each gen- eration step depends on the previously generated to- kens, achieve state-of-the-art (SOTA) performance on most datasets for machine translation tasks. AT model can better model the process of translation generation but leads to a massive limitation of its inference speed. Therefore, the non-autoregressive translation (NAT) (Gu et al., 2018) model is proposed, which is 15.6 times faster than AT model. NAT assumes that the generated tokens are conditionally independent 1Our code is released at https://github.com/ boom-R123/Candidate_Soups. Figure 1: Efficiency (Speedup) and Translation quality (BLEU) of NAT models in the WMT’14 EN-DE trans- lation dataset. A cross “×” represents our Candidate Soups (CDS) variants. Its base model is shown in the shape “•”, and its correspondence is represented by an arrow. CDS (mE-nD) refers to the AT model for re- scoring that has m encoder layers and n decoder layers. given the source sentence, so the translation can be generated in parallel, significantly improving its in- ference speed compared to AT. However, due to the strong independence assumption, the ability of the NAT model modeling sequence generation is weak- ened. So NAT model usually has multimodality problem (Gu et al., 2018) in the inference process, resulting in its performance worse than AT models. Several methods have been proposed to alleviate the multimodality problem and improve the perfor- mance of the NAT model, such as the iteration- based NAT model (Ghazvininejad et al., 2019; Gu et al., 2019; Kasai et al., 2020) and the semi- autoregressive translation model (Wang et al., 2018; Ran et al., 2020). Most …

Content truncated to 500 words