A Study of Non-autoregressive Model for Sequence Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 149–159 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 149 A Study of Non-autoregressive Model for Sequence Generation Yi Ren ∗ Zhejiang University rayeren@zju.edu.cn Jinglin Liu ∗ Zhejiang University jinglinliu@zju.edu.cn Xu Tan Microsoft Research Asia xuta@microsoft.com Zhou Zhao† Zhejiang University zhaozhou@zju.edu.cn Sheng Zhao Microsoft STC Asia Sheng.Zhao@microsoft.com Tie-Yan Liu Microsoft Research Asia tyliu@microsoft.com Abstract Non-autoregressive (NAR) models generate all the tokens of a sequence in parallel, re- sulting in faster generation speed compared to their autoregressive (AR) counterparts but at the cost of lower accuracy. Different tech- niques including knowledge distillation and source-target alignment have been proposed to bridge the gap between AR and NAR mod- els in various tasks such as neural machine translation (NMT), automatic speech recogni- tion (ASR), and text to speech (TTS). With the help of those techniques, NAR models can catch up with the accuracy of AR models in some tasks but not in some others. In this work, we conduct a study to understand the difficulty of NAR sequence generation and try to answer: (1) Why NAR models can catch up with AR models in some tasks but not all? (2) Why techniques like knowledge dis- tillation and source-target alignment can help NAR models. Since the main difference be- tween AR and NAR models is that NAR mod- els do not use dependency among target tokens while AR models do, intuitively the difficulty of NAR sequence generation heavily depends on the strongness of dependency among tar- get tokens. To quantify such dependency, we propose an analysis model called CoMMA to characterize the difficulty of different NAR se- quence generation tasks. We have several inter- esting findings: 1) Among the NMT, ASR and TTS tasks, ASR has the most target-token de- pendency while TTS has the least. 2) Knowl- edge distillation reduces the target-token de- pendency in target sequence and thus improves the accuracy of NAR models. 3) Source-target alignment constraint encourages dependency ∗ Equal contribution. † Corresponding author of a target token on source tokens and thus eases the training of NAR models. 1 Introduction Non-autoregressive (NAR) models (Oord et al., 2017; Gu et al., 2017; Chen et al., 2019; Ren et al., 2019), which generate all the tokens in a target sequence in parallel and can speed up inference, are widely explored in natural language and speech processing tasks such as neural machine transla- tion (NMT) (Gu et al., 2017; Lee et al., 2018; Guo et al., 2019a; Wang et al., 2019; Li et al., 2019b; Guo et al., 2019b), automatic speech recognition (ASR) (Chen et al., 2019) and text to speech (TTS) synthesis (Oord et al., 2017; Ren et al., 2019). How- ever, NAR models usually lead to lower accuracy than their autoregressive (AR) counterparts since the inner dependencies among the target tokens are explicitly removed. Several techniques have been proposed to allevi- ate the accuracy degradation, including 1) knowl- edge distillation (Oord et al., 2017; Gu …

Content truncated to 500 words