Conditional set generation using Seq2seq models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4874 - 4896 December 7-11, 2022 ©2022 Association for Computational Linguistics Conditional Set Generation Using SEQ2SEQ Models Aman Madaan, Dheeraj Rajagopal, Niket Tandon†, Yiming Yang, Antoine Bosselut‡ Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA † Allen Institute for Artificial Intelligence, Seattle, WA, USA ‡ EPFL, Switzerland amadaan@cs.cmu.edu Abstract Conditional set generation learns a mapping from an input sequence of tokens to a set. Sev- eral NLP tasks, such as entity typing and di- alogue emotion tagging, are instances of set generation. SEQ2SEQ models, a popular choice for set generation, treat a set as a sequence and do not fully leverage its key properties, namely order-invariance and cardinality. We propose a novel algorithm for effectively sampling infor- mative orders over the combinatorial space of label orders. We jointly model the set cardinal- ity and output by prepending the set size and taking advantage of the autoregressive factor- ization used by SEQ2SEQ models. Our method is a model-independent data augmentation ap- proach that endows any SEQ2SEQ model with the signals of order-invariance and cardinality. Training a SEQ2SEQ model on this augmented data (without any additional annotations) gets an average relative improvement of 20% on four benchmark datasets across various models: BART-base, T5-11B, and GPT3-175B.1 1 Introduction Conditional set generation is the task of modeling the distribution of an output set given an input se- quence of tokens (Kosiorek et al., 2020). Several NLP tasks are instances of set generation, including open-entity typing (Choi et al., 2018; Dai et al., 2021), fine-grained emotion classification (Dem- szky et al., 2020), and keyphrase generation (Meng et al., 2017; Yuan et al., 2020; Ye et al., 2021). The recent successes of the pretraining-finetuning paradigm have encouraged a formulation of set generation as a SEQ2SEQ generation task (Vinyals et al., 2016; Yang et al., 2018; Meng et al., 2019; Ju et al., 2020). In this paper, we posit that modeling set genera- tion as a vanilla SEQ2SEQ generation task is sub- optimal, because the SEQ2SEQ formulations do not 1Code to use SETAUG available at: https://setgen. structgen.com explicitly account for two key properties of a set output: order-invariance and cardinality. Forgo- ing order-invariance, vanilla SEQ2SEQ generation treats a set as a sequence, assuming an arbitrary order between the elements it outputs. Similarly, the cardinality of sets is ignored, as the number of elements to be generated is typically not modeled. Prior work has highlighted the importance of these two properties for set output through loss functions that encourage order invariance (Ye et al., 2021), exhaustive search over the label space for finding an optimal order (Qin et al., 2019; Rezatofighi et al., 2018; Vinyals et al., 2016), and post-processing the output (Nag Chowdhury et al., 2016). Despite the progress, several important gaps remain. First, exhaustive search does not scale with large output spaces typically found in NLP prob- lems, thus stressing the need for an optimal sam- pling strategy for the labels. Second, …

Content truncated to 500 words