Earlier Isn’t Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarization

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3324–3335, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3324 Earlier Isn’t Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarization Taehee Jung∗ ♥ Dongyeop Kang∗ ♣ Lucas Mentch ♥ Eduard Hovy ♣ ♥Department of Statistics, University of Pittsburgh, Pittsburgh, PA, USA ♣School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA {taj41,lkm31}@pitt.edu {dongyeok,hovy}@cs.cmu.edu Abstract Despite the recent developments on neural summarization systems, the underlying logic behind the improvements from the systems and its corpus-dependency remains largely un- explored. Position of sentences in the origi- nal text, for example, is a well known bias for news summarization. Following in the spirit of the claim that summarization is a combination of sub-functions, we define three sub-aspects of summarization: position, importance, and diversity and conduct an extensive analysis of the biases of each sub-aspect with respect to the domain of nine different sum- marization corpora (e.g., news, academic pa- pers, meeting minutes, movie script, books, posts). We find that while position exhibits substantial bias in news articles, this is not the case, for example, with academic papers and meeting minutes. Furthermore, our empirical study shows that different types of summariza- tion systems (e.g., neural-based) are composed of different degrees of the sub-aspects. Our study provides useful lessons regarding con- sideration of underlying sub-aspects when col- lecting a new summarization dataset or devel- oping a new system. 1 Introduction Despite numerous recent developments in neural summarization systems (Narayan et al., 2018b; Nallapati et al., 2016; See et al., 2017; Kedzie et al., 2018; Gehrmann et al., 2018; Paulus et al., 2017) the underlying rationales behind the im- provements and their dependence on the training corpus remain largely unexplored. Edmundson (1969) put forth the position hypothesis: impor- tant sentences appear in preferred positions in the document. Lin and Hovy (1997) provide a method to empirically identify such positions. Later, Hong and Nenkova (2014) showed an intentional lead ∗ Equal contribution, name order decided by coin flip. bias in news writing, suggesting that sentences appearing early in news articles are more impor- tant for summarization tasks. More generally, it is well known that recent state-of-the-art models (Nallapati et al., 2016; See et al., 2017) are of- ten marginally better than the first-k baseline on single-document news summarization. In order to address the position bias of news articles, Narayan et al. (2018a) collected a new dataset called XSum to create single sentence summaries that include material from multiple po- sitions in the source document. Kedzie et al. (2018) showed that the position bias in news ar- ticles is not the same across other domains such as meeting minutes (Carletta et al., 2005). In addition to position, Lin and Bilmes (2012) defined other sub-aspect functions of sum- marization including coverage, diversity, and information. Lin and Bilmes (2011) claim that many existing summarization systems are instances of mixtures of such …

Content truncated to 500 words