Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10834 - 10858 December 7-11, 2022 ©2022 Association for Computational Linguistics Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention Wenhao Li1,2,3, Xiaoyuan Yi5, Jinyi Hu1,2,3, Maosong Sun1,2,3,4∗, Xing Xie5 1 Department of Computer Science and Technology, Tsinghua University, Beijing 2 Beijing National Research Center for Information Science and Technology 3 Institute for Artificial Intelligence, Tsinghua University, Beijing 4 Jiangsu Collaborative Innovation Center for Language Ability, Jiangsu Normal University, Xuzhou 5 Microsoft Research Asia wh-li20@mails.tsinghua.edu.cn, xiaoyuanyi@microsoft.com Abstract Recently, powerful Transformer architectures have proven superior in generating high-quality sentences. Nevertheless, these models tend to produce dull high-frequency phrases, severely hurting the diversity and novelty of generated text. In this work, we dig into the intrinsic mechanism of this problem and found that sparser attention values in Transformer could improve diversity. To understand such a phe- nomenon, we first conduct both empirical and theoretical analysis and then attribute it to rep- resentation degeneration caused by the atten- tive mixture of the hidden states during train- ing. We term this process the Trap of Medi- ocrity. To escape from such a trap, we intro- duce a novel attention regularization loss to control the sharpness of the attention distribu- tion, which is transparent to model structures and can be easily implemented within 20 lines of python code. We prove that this method could be mathematically regarded as learning a Bayesian approximation of posterior attention. Experiments show that our method improved the diversity and novelty of the generated text while maintaining comparable quality on a vari- ety of conditional and unconditional generation tasks. 1 Introduction Natural Language Generation (NLG) plays a cru- cial role in modern natural language Processing (NLP). With considerable benefits for a wide ar- ray of applications such as automatic literary writ- ing (Yi et al., 2018; Ao et al., 2021), summariza- tion (Huang et al., 2021) and paraphasing (Lin et al., 2021), this area has risen to prominence. Recently, neural architectures based on the preva- lent Tranformer (Vaswani et al., 2017), e.g., GPT-2 (Radford et al., 2019), have proven highly success- ful in producing fluent text indistinguishable from the human-written one. Nevertheless, these models ∗Corresponding author. Email: sms@tsinghua.edu.cn 2 4 8 16 32 64 128 320 Window Size 83.0 83.1 83.2 83.3 83.4 83.5 83.6 83.7 83.8 83.9 Quality 25.4 25.6 25.8 26.0 26.2 26.4 26.6 26.8 27.0 27.2 Diversity Figure 1: Quality and diversity of generated text with varying attention sparsity on ROCStories. The qual- ity and diversity (the higher the better) are measured by BERTScore (Zhang et al., 2020) and dist (Li et al., 2016a), respectively. Sparsity is controlled by the at- tention window size. A smaller window leads to higher sparsity. tend to remember and generate dull and generic (usually high-frequency) contents (Holtzman et al., 2020), even with highly-diverse ground-truth as training targets and distinct prompts as testing in- puts. Such a conundrum, known as one kind of degeneration …

Content truncated to 500 words