JANUS: Joint Autoregressive and Non-autoregressive Training with Auxiliary Loss for Sequence Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8050 - 8060 December 7-11, 2022 ©2022 Association for Computational Linguistics JANUS: Joint Autoregressive and Non-autoregressive Training with Auxiliary Loss for Sequence Generation Xiaobo Liang1 Juntao Li1 Lijun Wu2 Min Zhang1 1Soochow University, 2Microsoft Research xbliang3@stu.suda.edu.cn, {ljt,minzhang}@suda.edu.cn lijuwu@microsoft.com Abstract Transformer-based autoregressive and non- autoregressive models have played an essen- tial role in sequence generation tasks. The autoregressive model can obtain excellent performance, while the non-autoregressive model brings fast decoding speed for infer- ence. In this paper, we propose JANUS, a Joint Autoregressive and Non-autoregressive training method using aUxiliary losS to en- hance the model performance in both AR and NAR manner simultaneously and effectively al- leviate the problem of distribution discrepancy. Further, we pre-train BART with JANUS on a large corpus with minimal cost (16 GPU days) and make the BART-JANUS capable of non- autoregressive generation, demonstrating that our approach can transfer the AR knowledge to NAR. Empirically, we show our approach and BART-JANUS can achieve significant improve- ment on multiple generation tasks, including machine translation and GLGE benchmarks. Our code is available at Github1. 1 Introduction The transformer-based autoregressive (Vaswani et al., 2017; So et al., 2019) (AR) generation model has achieved high-quality results in various natural language generation tasks. Meanwhile, the non- autoregressive (NAR) (Gu et al., 2018; Lee et al., 2018; Libovick`y and Helcl, 2018; Su et al., 2021) generation methods show great potential to reduce the inference latency by introducing parallel decod- ing. Especially, iterative generative paradigms like CMLM (Wang et al., 2019a; Ghazvininejad et al., 2019) can dynamically adapt the trade-off between performance and latency. Recent works (Qi et al., 2021; Guo et al., 2020; Tian et al., 2020) have suc- cessfully combined these two mechanisms by joint training. However, these approaches only consider the relevance of model parameters, ignoring the cor- relations between the two manners, which require 1https://github.com/dropreg/JANUS Figure 1: The example of JANUS. We exhibit some cases that show the distribution discrepancy due to the difference in context with orange background. efforts for improvement. Therefore, in this paper, we attempt to leverage the merits of both AR and NAR mechanisms while avoiding their weaknesses to improve their performance. The principal difference between AR and NAR is that they use distinct attention mechanisms. In particular, the AR uses unidirectional atten- tion (Vaswani et al., 2017) to simplify the sentence probability distributions by introducing Markov Hypothesis, which outputs the next token only depending on the previous context. This pattern makes the output distribution of each token accu- rate and unambiguous but also lacks diversity in inference. The NAR introduces bi-directional at- tention (Devlin et al., 2019) that can capture bi- directional context by considering the whole sen- tence information. This pattern needs to predict multiple tokens simultaneously, which causes the token distribution to be ambiguous, stemming from the multi-modality problem (Zhou et al., 2019). Inspired by knowledge distillation (Hinton et al., 2015) and deep mutual learning (Zhang et al., 2018b), we …

Content truncated to 500 words