POS-Constrained Parallel Decoding for Non-autoregressive Generation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5990–6000 August 1–6, 2021. ©2021 Association for Computational Linguistics 5990 POS-Constrained Parallel Decoding for Non-autoregressive Generation Kexin Yang♠ Wenqiang Lei♦∗ Dayiheng Liu♠ Weizhen Qi♣ Jiancheng Lv♠ ♠College of Computer Science, Sichuan University ♦National University of Singapore ♣University of Science and Technology of China {kexinyang0528, wenqianglei}@gmail.com Abstract The multimodality problem has become a ma- jor challenge of existing non-autoregressive generation (NAG) systems. A common solu- tion often resorts to sequence-level knowledge distillation by rebuilding the training dataset through autoregressive generation (hereinafter known as “teacher AG”). The success of such methods may largely depend on a latent as- sumption, i.e., the teacher AG is superior to the NAG model. However, in this work, we exper- imentally reveal that this assumption does not always hold for the text generation tasks like text summarization and story ending genera- tion. To provide a feasible solution to the mul- timodality problem of NAG, we propose incor- porating linguistic structure (Part-of-Speech sequence in particular) into NAG inference in- stead of relying on teacher AG. More specif- ically, the proposed POS-constrained Parallel Decoding (POSPD) method aims at provid- ing a specific POS sequence to constrain the NAG model during decoding. Our experi- ments demonstrate that POSPD consistently improves NAG models on four text generation tasks to a greater extent compared to knowl- edge distillation. This observation validates the necessity of exploring the alternatives for sequence-level knowledge distillation. 1 Introduction Unlike autoregressive generation (AG) that gener- ates tokens step-by-step, non-autoregressive gener- ation (NAG) parallelly generates all tokens in one time step and thus the inference could be signifi- cantly speeded up (Ma et al., 2019; Ran et al., 2020; Susanto et al., 2020). Despite the computational advantage of NAG, it has faced the multimodality problem (Gu et al., 2018) caused by the condition- ally independent decoding. A typical example of the problem is illustrated in Figure 1, where either ∗ Correspondence to Wenqiang Lei. Figure 1: An example to explain “multimodality prob- lem”. The German sentence “Vielen Dank.” can be translated into “Many Thanks.” and “Thank you.”. of “Thank you.” and “Many Thanks.” is the correct translation (i.e., generation modes). In this exam- ple, a mixed mode “Many you.” / “Thank Thanks.” will be generated by NAG. It is because the con- ditional dependence among target words will be broken in parallel decoding. A typical manifesta- tion is that words are usually missing (e.g., “Many you.”) and repeating (e.g., “Thank Thanks.”) in NAG’s sentences. To solve this problem, the key is helping NAG models to deal with various genera- tion modes. To date, one of the most widely used solutions is sequence-level knowledge distillation (Kim and Rush, 2016) which aims to reduce the generation modes of the raw data (Zhou et al., 2019). Tak- ing machine translation as an example, the knowl- edge distillation based methods rebuild the target sequence in the training set by employing …

Content truncated to 500 words