Attention Head Masking for Inference Time Content Selection in Abstractive Summarization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5008–5016 June 6–11, 2021. ©2021 Association for Computational Linguistics 5008 Attention Head Masking for Inference Time Content Selection in Abstractive Summarization Shuyang Cao and Lu Wang Computer Science and Engineering University of Michigan Ann Arbor, MI {caoshuy, wangluxy}@umich.edu Abstract How can we effectively inform content selec- tion in Transformer-based abstractive summa- rization models? In this work, we present a simple-yet-effective attention head mask- ing technique, which is applied on encoder- decoder attentions to pinpoint salient content at inference time. Using attention head mask- ing, we are able to reveal the relation between encoder-decoder attentions and content selec- tion behaviors of summarization models. We then demonstrate its effectiveness on three doc- ument summarization datasets based on both in-domain and cross-domain settings. Impor- tantly, our models outperform prior state-of- the-art models on CNN/Daily Mail and New York Times datasets. Moreover, our inference- time masking technique is also data-efficient, requiring less than 20% of the training sam- ples to outperform BART fine-tuned on the full CNN/DailyMail dataset. 1 Introduction Large pre-trained Transformers have achieved state- of-the-art results on various summarization datasets with a fine-tuning phase to streamline the summa- rization pipeline (Lewis et al., 2020; Yan et al., 2020). Yet, it is still unclear how one can use large models more effectively for abstractive sum- marization . For example, prior work shows that informing content selection via attention weight updating in recurrent neural networks can further boost summarizer performance (Gehrmann et al., 2018). However, with multi-heads attentions at all layers in Transformers (Vaswani et al., 2017), highlighting salient content becomes non-trivial. In this work, we propose an inference-time at- tention head masking mechanism that works on encoder-decoder attentions to underscore salient content from the source and improve the quality of abstractive summaries. Based on this mechanism, we first demonstrate the relation between encoder- decoder attentions and content selection behaviors, on three summarization datasets of CNN/DailyMail (CNN/DM), New York Times (NYT), and XSum. Second, we study whether multiple heads at the same layer collectively guide the summarization. Partial masking is found to be most effective, indi- cating a strong collaborative effect and the impor- tance of head selection. Based on these observations, we evaluate at- tention head masking on summarization bench- marks with salience labels provided by externally trained content selectors. On all three datasets, our model consistently outperforms fine-tuned BART (Lewis et al., 2020) and several top per- forming Transformer-based abstractive summariza- tion models (Zhang et al., 2019b; Yan et al., 2020). Summaries generated by our model are also con- sidered to have better informativeness by human judges. Moreover, we illustrate that attention head masking is data-efficient: on CNN/DM, BART fine-tuned on less than 20% of the training data outperforms a version trained on the full set. Fi- nally, we show that our method is effective under a cross-domain setting. With a content selector trained on NYT, BART fine-tuned on CNN/DM gains more …

Content truncated to 500 words