Diformer: Directional Transformer for Neural Machine Translation

Diformer: Directional Transformer for Neural Machine Translation Minghan Wang1, Jiaxin Guo1, Yuxia Wang2, Daimeng Wei1, Hengchao Shang1, Yinglu Li1, Chang Su1, Yimeng Chen1, Min Zhang1, Shimin Tao1, Hao Yang1 1Huawei Translation Services Center, Beijing, China 2The University of Melbourne, Melbourne, Australia {wangminghan,guojiaxin1,weidaimeng,shanghengchao, liyinglu,suchang8,chenyimeng,zhangmin186, taoshimin,yanghao30}@huawei.com yuxiaw@student.unimelb.edu.au Abstract Autoregressive (AR) and Non- autoregressive (NAR) models have their own superiority on the performance and latency, combining them into one model may take advantage of both. Cur- rent combination frameworks focus more on the integration of multiple decoding paradigms with a unified generative model, e.g. Masked Language Model. However, the generalization can be harm- ful on the performance due to the gap between training objective and inference. In this paper, we aim to close the gap by preserving the original objective of AR and NAR under a unified framework. Specifically, we propose the Directional Transformer (Diformer) by jointly mod- elling AR and NAR into three generation directions (left-to-right, right-to-left and straight) with a newly introduced direction variable, which works by controlling the prediction of each token to have specific dependencies under that direction. The unification achieved by direction success- fully preserves the original dependency assumption used in AR and NAR, retain- ing both generalization and performance. Experiments on 4 WMT benchmarks demonstrate that Diformer outperforms current united-modelling works with more than 1.5 BLEU points for both AR and NAR decoding, and is also competitive to the state-of-the-art independent AR and NAR models. © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1 Introduction Machine translation can be considered as a condi- tional generation task, which has been dominated by neural networks, especially after Transformer (Vaswani et al., 2017). Conventional autoregres- sive (AR) NMT models obtain the impressive per- formance, but it’s time-consuming to decode token one by one sequentially (Sutskever et al., 2014; Bahdanau et al., 2015). Aiming at fast inference, non-autoregressive (NAR) NMT models enhance the parallelizability by reducing or removing the sequential dependency on the translation prefix in- side the decoder, but suffering from performance degradation owing to the multi-modality problem, which is still an open-question (Gu et al., 2018; Shu et al., 2020; Ghazvininejad et al., 2020; Lee et al., 2018; Ghazvininejad et al., 2019; Stern et al., 2019; Welleck et al., 2019; Gu et al., 2019a; Gu et al., 2019b). It’s always non-trivial to balance high perfor- mance and low latency in a single model perfectly. Therefore, another branch focuses on the unified- modeling of multiple decoding paradigms so that decoding with AR or NAR in different scenar- ios (AR for quality-first and NAR for speed-first) with one model can be achieved (Mansimov et al., 2020; Tian et al., 2020; Qi et al., 2021), making the performance and speed can be pursued more practically. Whereas, challenges still exist. For example, a generalized conditional language model is often required to support the generation with customized orders or positions (Mansimov et al., 2020; Tian et al., 2020), which actually prevents the …

Content truncated to 500 words