SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2450 - 2458 July 10-15, 2022 ©2022 Association for Computational Linguistics SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling Forrest Sheng Bao and Ge Luo and Hebi Li Iowa State University, Ames, IA, USA forrest.bao@gmail.com, {gluo,hebi}@iastate.edu Minghui Qiu Alibaba Group Hangzhou, China minghui.qmh@alibaba-inc.com Yinfei Yang 1600 Amphitheater Parkway Mountain View, CA, USA yangyin7@gmail.com Youbiao He Iowa State University Ames, IA, USA yh54@iastate.edu Cen Chen East China Normal University Shanghai, China cecilia.cenchen@gmail.com Abstract Canonical automatic summary evaluation met- rics, such as ROUGE, focus on lexical simi- larity which cannot well capture semantics nor linguistic quality and require a reference sum- mary which is costly to obtain. Recently, there have been a growing number of efforts to al- leviate either or both of the two drawbacks. In this paper, we present a proof-of-concept study to a weakly supervised summary evalua- tion approach without the presence of reference summaries. Massive data in existing summa- rization datasets are transformed for training by pairing documents with corrupted reference summaries. In cross-domain tests, our strategy outperforms baselines with promising improve- ments, and show a great advantage in gauging linguistic qualities over all metrics. 1 Introduction In natural language processing, the problem of sum- marization studies generating a summary from a source document which is longer than the summary. De facto metrics to judge a generated summary in- clude ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), and METEOR (Banerjee and Lavie, 2005). Previous work (Ng and Abrecht, 2015; Liu and Liu, 2008; Liu et al., 2016; Shang et al., 2018) agrees on two major drawbacks of them: 1) they favor lexical similarity, falling short on semantic similarity or linguistic quality, and 2) they require a reference summary which is often expensive to obtain (Zopf, 2018). Initially, the first drawback is partially allevi- ated by replacing lexicons with their word embed- dings (Ng and Abrecht, 2015; Ellouze et al., 2017; Ruseti et al., 2018; Xia et al., 2019). After the birth of transformers (Vaswani et al., 2017), this effort has expanded to sentence or document level, including reference-based (Zhang* et al., 2020; Zhao et al., 2019), and reference-free ones (Vasi- lyev et al., 2020; Scialom et al., 2019; Gao et al., 2020). The main difference between the two groups is whether a reference summary is needed when evaluating a machine-generated summary. The two groups have complementary pros and cons. Reference-based metrics have a better perfor- mance, but they are impractical when summariza- tion is used industrially, such as in customer sup- port (Liu et al., 2019), team conversation (Zhang and Cranshaw, 2018), and bug reporting (Rastkar et al., 2014), where it is too costly to manually craft an equally massive amount of reference summaries. In contrast, without human written reference sum- maries, reference-free approaches generally per- form poorer. Modern transformer-based reference- free approaches often rely on non-summarization tasks, such as QA (Vasilyev et …

Content truncated to 500 words