A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1319 - 1339 May 22-27, 2022 c⃝2022 Association for Computational Linguistics A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation Shashi Narayan Google Research shashinarayan@google.com Gonçalo Simoes Google Research gsimoes@google.com Yao Zhao Google Brain yaozhaoyz@google.com Joshua Maynez Google Research joshuahm@google.com Dipanjan Das Google Research dipanjand@google.com Michael Collins Google Research mjcollins@google.com Mirella Lapata Google Research lapata@google.com Abstract We propose Composition Sampling, a sim- ple but effective method to generate diverse outputs for conditional generation of higher quality compared to previous stochastic decod- ing strategies. It builds on recently proposed plan-based neural generation models (Narayan et al., 2021) that are trained to first create a composition of the output and then generate by conditioning on it and the input. Our approach avoids text degeneration by first sampling a composition in the form of an entity chain and then using beam search to generate the best possible text grounded to this entity chain. Ex- periments on summarization (CNN/DailyMail and XSum) and question generation (SQuAD), using existing and newly proposed automatic metrics together with human-based evaluation, demonstrate that Composition Sampling is cur- rently the best available decoding strategy for generating diverse meaningful outputs. 1 Introduction In many NLG tasks, it is important to be able to gen- erate multiple diverse outputs from a model. Tasks like summarization (Mani, 2001; Nenkova and McKeown, 2011) and question generation (Zhou et al., 2017) exhibit one-to-many relationships; there can be multiple semantically diverse sum- maries or questions for the same source, and it may be useful for a model to be able to generate multiple outputs. Yet, the primary focus of recent research in NLG has been on improving the quality of single-best outputs (Raffel et al., 2019; Lewis et al., 2019; Dong et al., 2019; Zhang et al., 2020a; Narayan et al., 2021), while diversity remains an unsolved problem (Hashimoto et al., 2019; Zhang et al., 2021). This is particularly challenging in con- ditional generation, where diversity in the target sequence should not come at the cost of correctness or faithfulness; for example, alternate summaries are not valuable if they are unfaithful to the input document(s) (Maynez et al., 2020; Kryscinski et al., 2020). In this work, we investigate decoding meth- ods for generating semantically diverse text which is also faithful to its input focusing on two tasks, namely summarization and question generation. Beam search (Li et al., 2016; Wiseman et al., 2017) has proven successful for single-best gen- eration (Rush et al., 2015; Barrault et al., 2020; Meister et al., 2020), but struggles to generate di- verse output (Vijayakumar et al., 2016). Stochastic sampling strategies, such as top-k sampling (Fan et al., 2018) and nucleus sampling (Holtzman et al., 2020), are better at generating diverse sequences but are not suitable for conditional generation as they degenerate,1 producing output that is not faith- ful to the source. Figure 1 exposes degeneration in summary output using nucleus sampling. …

Content truncated to 500 words