On-the-Fly Attention Modulation for Neural Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1261–1274 August 1–6, 2021. ©2021 Association for Computational Linguistics 1261 On-the-Fly Attention Modulation for Neural Generation Yue Dong1∗ Chandra Bhagavatula2 Ximing Lu2,4 Jena D. Hwang2 Antoine Bosselut2,3 Jackie Chi Kit Cheung1 Yejin Choi2,4 1Mila / McGill University 2Allen Institute for Artificial Intelligence 3 Stanford University 4Paul G. Allen School of CSE, University of Washington {yue.dong2@mail, jcheung@cs}.mcgill.ca {chandrab,ximinglu,jenah,antoineb,yejinc}@allenai.org Abstract Despite considerable advancements with deep neural language models (LMs), neural text generation still suffers from degeneration: the generated text is repetitive, generic, self- contradictory, and often lacks commonsense. Our analyses on sentence-level attention pat- terns in LMs reveal that neural degeneration may be associated with insufficient learning of task-specific characteristics by the atten- tion mechanism. This finding motivates on- the-fly attention modulation1– a simple but ef- fective method that enables the injection of priors into attention computation during infer- ence. Automatic and human evaluation results on three text generation benchmarks demon- strate that attention modulation helps LMs gen- erate text with enhanced fluency, creativity, and commonsense reasoning, in addition to significantly reduce sentence-level repetition. 1 Introduction Neural text generation is critical for a wide range of downstream natural language applications. How- ever, the standard approach – using a Transformer- based (Vaswani et al., 2017) language model (e.g., Radford et al., 2019) with maximum likelihood fine-tuning and non-stochastic decoding – is known to exhibit degeneration (Welleck et al., 2019). De- spite being pre-trained on large amounts of data, text generated by neural models is observed to be repetitive, generic, self-contradictory, and lacking commonsense (Holtzman et al., 2020). Many explanations have been proposed for neu- ral text degeneration, including inappropriate train- ing objectives (Welleck et al., 2019) and decoding discrepancies relative to human language (Holtz- man et al., 2018, 2020). While the aforementioned may be factors for neural degeneration, we show ∗*This work was done when the first author was an intern at AI2. Figure 1: Example of fine-tuned GPT2-L outputs with- out (top) and with (bottom) attention modulation on αNLG. The task is to generate a plausible explana- tory hypothesis H for observations O1 and O2. Our proposed attention modulation injects the task-specific prior – LMs should consider both observations rela- tive equally – through balancing the sentence-level at- tention weights (Eqn. 5) in Transformer blocks dur- ing inference. Applying attention modulation with the aforementioned prior make sentence-level attentions from generation to observation pairs (O1,O2) more bal- anced2, which are reflected in the sentence-level atten- tion heatmaps of GPT2-L (darker = lower attention) across layers (y-axis) and heads (x-axis). that insufficient learning of task-specific character- istics – reflected in the self-attention mechanism in transformer blocks – is associated with neural text degeneration. We demonstrate that degeneration is alleviated if we inject priors through attention modulation (AttnM) during inference. Self-attention – the ubiquitous component of Transformers – is task-agnostic with a large learn- ing capacity for many NLP tasks (Vaswani et al., 2017; Devlin et al., 2019; Brown et al., 2020). It 2attention ratios …

Content truncated to 500 words