Extract, Denoise and Enforce: Evaluating and Improving Concept Preservation for Text-to-Text Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5063–5074 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5063 Extract, Denoise and Enforce: Evaluating and Improving Concept Preservation for Text-to-Text Generation Yuning Mao1, Wenchang Ma2⋆, Deren Lei3⋆, Jiawei Han1, Xiang Ren3 1University of Illinois at Urbana-Champaign 2Tsinghua University 3University of Southern California 1{yuningm2, hanj}@illinois.edu 2mwc17@mails.tsinghua.edu.cn 3{derenlei, xiangren}@usc.edu Abstract Prior studies on text-to-text generation typi- cally assume that the model could figure out what to attend to in the input and what to in- clude in the output via seq2seq learning, with only the parallel training data and no addi- tional guidance. However, it remains unclear whether current models can preserve impor- tant concepts in the source input, as seq2seq learning does not have explicit focus on the concepts and commonly used evaluation met- rics also treat concepts equally important as other tokens. In this paper, we present a systematic analysis that studies whether cur- rent seq2seq models, especially pre-trained language models, are good enough for preserv- ing important input concepts and to what ex- tent explicitly guiding generation with the con- cepts as lexical constraints is beneficial. We answer the above questions by conducting ex- tensive analytical experiments on four repre- sentative text-to-text generation tasks. Based on the observations, we then propose a sim- ple yet effective framework to automatically extract, denoise, and enforce important in- put concepts as lexical constraints. This new method performs comparably or better than its unconstrained counterpart on automatic met- rics, demonstrates higher coverage for concept preservation, and receives better ratings in the human evaluation.1 1 Introduction Text-to-text generation is an important research problem with a broad set of applications, such as dialog response generation (Dinan et al., 2019), headline generation (Gu et al., 2020), and sum- marization (Mao et al., 2020b). A distinct feature of text-to-text generation (vs. free-form text gen- eration) is that it is often desired to preserve the *Equal contribution 1Our code is available at https://github.com/ morningmoni/EDE [Target Output] What year did Nintendo announce a new Legend of Zelda was in the works for Gamecube? [Source Input] In 2003, Nintendo announced that a new The Legend of Zelda game was in the works for the GameCube by the same team that had created the cel- shaded The Wind Waker. … [SEP] 2003 [Question Generation] [Target Output] Warner Bros. is actually in Burbank, which I think is close to Hollywood. Do you know how Hollywood got its name? It's a pretty funny story! [Source Input] Knowledge: Warner Bros. Entertainment Inc. (formerly Warner Brothers Pictures, Inc.) is an American entertainment company that is a division of Time Warner and is headquartered in Burbank, California. Context: ... Is Warner Bros. based in Hollywood? [Knowledge-Grounded Dialog] [Target Output] Israel launches third airstrike on Gaza city. [Source Input] Israel's air forces on Monday launched a third missile strike on Gaza city in less than four hours, Palestinian witnesses said. [Headline Generation] Figure 1: Examples of text-to-text generation tasks where preserving important input …

Content truncated to 500 words