Noisy Self-Knowledge Distillation for Text Summarization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 692–703 June 6–11, 2021. ©2021 Association for Computational Linguistics 692 Noisy Self-Knowledge Distillation for Text Summarization Yang Liu♥, Sheng Shen♦, and Mirella Lapata△ ♥Microsoft Cognitive Services Research ♦University of California, Berkeley △School of Informatics, University of Edinburgh yaliu10@microsoft.com sheng.s@berkeley.edu mlap@inf.ed.ac.uk Abstract In this paper we apply self-knowledge dis- tillation to text summarization which we ar- gue can alleviate problems with maximum- likelihood training on single reference and noisy datasets. Instead of relying on one-hot annotation labels, our student summarization model is trained with guidance from a teacher which generates smoothed labels to help reg- ularize training. Furthermore, to better model uncertainty during training, we introduce mul- tiple noise signals for both teacher and student models. We demonstrate experimentally on three benchmarks that our framework boosts the performance of both pretrained and non- pretrained summarizers achieving state-of-the- art results.1 1 Introduction Automatic summarization has enjoyed renewed in- terest in recent years, thanks to the popularity of neural network models and their ability to learn continuous representations without recourse to pre- processing tools or linguistic annotations. The availability of large-scale datasets (Sandhaus, 2008; Hermann et al., 2015; Grusky et al., 2018; Narayan et al., 2018) containing hundreds of thousands of document-summary pairs has driven the develop- ment of neural architectures for summarization. Several approaches have been proposed, in the vast majority sequence-to-sequence models which are trained in an end-to-end fashion with a maximum likelihood estimation loss (See et al., 2017; Celiky- ilmaz et al., 2018; Paulus et al., 2018; Gehrmann et al., 2018). Despite promising results, there are specific char- acteristics of the summarization task which ren- der it ill-suited to standard sequence-to-sequence training. For instance, maximum-likelihood train- ing on single reference datasets might not be opti- mal for summarization which is subject to a great 1Our code is available at https://github.com/ nlpyang/NoisySumm. deal of human variation (Harman and Over, 2004; Nenkova, 2006). In the context of extractive sum- marization, different people select different sen- tences to include in a summary (Rath et al., 1961), and when writing abstracts, disagreement exists both in terms of writing style and the specific con- tent deemed important for the summary (Harman and Over, 2004). Although summarization models would naturally benefit from multiple target refer- ences, it is unrealistic to expect that multi-reference datasets can be created at scale for neural network training. In fact, most popular benchmarks are col- lated opportunistically, based on summaries which only loosely correspond to the source input. For example, Narayan et al. (2018) create a dataset by pairing the first sentence of a news article with the rest of the document under the assump- tion that the introductory sentence expresses the gist of the article. Grusky et al. (2018) pair arti- cles with metadata available in HTML pages under the assumption that HTML tags (e.g., description) denote summary-like content. In other work (Liu et al., 2018; Perez-Beltrachini et al., 2019), …

Content truncated to 500 words