Corpora Evaluation and System Bias Detection in Multi-document Summarization

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2830–2840 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2830 Corpora Evaluation and System Bias Detection in Multi-document Summarization Alvin Dey†∗ Tanya Chowdhury♦†∗ Yash Kumar Atri†∗ Tanmoy Chakraborty† ♦University of Massachusetts, Amherst, MA, USA †IIIT-Delhi, New Delhi, India tchowdhury@cs.umass.edu {alvin18066,yashk,tanmoy}@iiitd.ac.in Abstract Multi-document summarization (MDS) is the task of reflecting key points from any set of documents into a concise text paragraph. In the past, it has been used to aggregate news, tweets, product reviews, etc. from various sources. Owing to no standard definition of the task, we encounter a plethora of datasets with varying levels of overlap and conflict be- tween participating documents. There is also no standard regarding what constitutes sum- mary information in MDS. Adding to the chal- lenge is the fact that new systems report results on a set of chosen datasets, which might not correlate with their performance on the other datasets. In this paper, we study this hetero- geneous task with the help of a few widely used MDS corpora and a suite of state-of-the- art models. We make an attempt to quantify the quality of summarization corpus and pre- scribe a list of points to consider while propos- ing a new MDS corpus. Next, we analyze the reason behind the absence of an MDS system which achieves superior performance across all corpora. We then observe the extent to which system metrics are influenced, and bias is propagated due to corpus properties. The scripts to reproduce the experiments in this work are available at https://github.com/ LCS2-IIITD/summarization_bias.git. 1 Introduction Multi-document summarization (MDS) deals with compressing more than one document into a tex- tual summary. It has a wide range of applications – gaining insights from tweets related to similar hashtags, understanding product features amongst e-commerce reviews, summarizing live blogs re- lated to an ongoing match, etc. Most studies on MDS were performed during the DUC1 and TAC2 challenges starting in the early 2000s. Each version of the challenges released a new dataset. Most of the MDS systems submitted to these challenges ∗Equal contribution; listed alphabetically. 1https://duc.nist.gov/ 2http://tac.nist.gov were unsupervised and extractive in nature. Gradu- ally, the data released in these challenges became the de facto for MDS. These datasets were manu- ally curated and had less than a hundred instances each. The recent development of deep neural ar- chitecture has led to a significant increase in the number of supervised document summarization systems. Large labeled corpora which are mostly crowd-sourced have been introduced to meet the training requirements of the supervised systems. However, the crowd-sourced corpora widely differ in quality based on factors like genre, size of the community, presence of moderation in the commu- nity, etc. This is further aggravated by the com- plexity of the task, the hardness of accumulating labeled data, or more so in the definition of what constitutes a multi-document summary. Recently, a few large datasets for MDS have been introduced (Fabbri et al., 2019; Chowdhury and Chakraborty, 2019a). …

Content truncated to 500 words