The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 187–196 November 10–11, 2021. ©2021 Association for Computational Linguistics 187 The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21 Lihua Qian∗ †, Yi Zhou∗†, Zaixiang Zheng∗†, Yaoming Zhu†, Zehui Lin†, Jiangtao Feng†, Shanbo Cheng†, Lei Li‡, Mingxuan Wang† and Hao Zhou† †Bytedance AI Lab ‡University of California Santa Barbara {qianlihua,zhouyi.naive,zhengzaixiang,zhuyaoming,linzehui}@bytedance.com {fengjiangtao,chengshanbo,wangmingxuan.89,zhouhao.nlp}@bytedance.com lilei@cs.ucsb.edu Abstract This paper describes the Volctrans’ submission to the WMT21 news translation shared task for German→English translation. We build a parallel (i.e., non-autoregressive) translation system using the Glancing Transformer (Qian et al., 2020), which enables fast and accu- rate parallel decoding in contrast to the cur- rently prevailing autoregressive models. To the best of our knowledge, this is the first par- allel translation system that can be scaled to such a practical scenario like WMT competi- tion. More importantly, our parallel translation system achieves the best BLEU score (35.0) on German→English translation task, outper- forming all strong autoregressive counterparts. 1 Introduction In recent years’ WMT competitions, most teams develop their translation systems based on autore- gressive models, such as Transformer (Vaswani et al., 2017). Although autoregressive mod- els (AT) achieve strong results, it is also worth exploring other alternative machine translation paradigm. Therefore, we build our systems with non-autoregressive translation (NAT) models (Gu et al., 2018). Unlike the left-to-right decoding in the autoregressive models, the NAT models employ the more efficient parallel decoding. Specifically, our system employs single-pass parallel decoding, which generates all the tokens in parallel at one time, thus can accelerate decoding speed. In this paper, we would like to present the best practice we explored in this year’s competition for our parallel translation system, aiming at achieving top results while preserving decoding efficiency. System Overview. To achieve this, we improve the parallel translation system in several aspects, including better model architectures, various data ∗Equal contributions. exploitation methods, mutli-stage training strategy, and inference with effective reranking techniques. For model architectures (§2), we build the parallel translation system based on the Glancing Trans- former (GLAT, Qian et al., 2020). Besides, our system employs dynamic linear combination of layers (DLCL, Wang et al., 2019) for training deep models. For data exploitation (§3), we first filter data with multiple strategies. After filtering, we use the Transformer (Vaswani et al., 2017) to syn- thesize various distilled data. For training (§4), the NAT models employ multi-stage training to better exploit the distilled data. At inference phase (§5), the system generates the final results by reranking candidate hypothesis from multiple parallel gener- ation models. With the proposed techniques, our parallel trans- lation system surpasses autoregressive models, and achieves the highest BLEU score (35.0) in the German→English translation task. Such results show that parallel translation system not only has great decoding efficiency, but also could achieve better performance compared to the autoregresssive counterparts. 2 Backbone Model Architecture As depicted in Figure 1, our submitted system em- ploys GLAT (Qian et al., 2020) as our backbone model architecture, and includes an auxiliary de- coder in …

Content truncated to 500 words