Entity-level Factual Consistency of Abstractive Text Summarization

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2727–2733 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2727 Entity-level Factual Consistency of Abstractive Text Summarization Feng Nan1 Ramesh Nallapati1 Zhiguo Wang1 Cicero Nogueira dos Santos1 Henghui Zhu1 Dejiao Zhang1 Kathleen McKeown1,2 Bing Xiang1 Amazon Web Services1, Columbia University2 {nanfen,rnallapa,zhiguow,cicnog,henghui,dejiaoz,mckeownk,bxiang} @amazon.com Abstract isA key challenge for abstractive summarization ensuring factual consistency of the generated summary with respect to the original document. For example, state-of- the-art models trained on existing datasets exhibit entity hallucination, generating names of entities that are not present in the source document. We propose a set of new metrics to quantify the entity-level factual consistency of generated summaries and we show that the entity hallucination problem can be alleviated by simply filtering the training data. In addition, we propose a summary-worthy entity classification task to the training process as well as a joint entity and summary generation approach, which yield further improvements in entity level metrics. 1 Introduction Many recent advances in deep neural networks have led to significant improvement in the quality of abstractive summarization (Radford et al., 2019; Gehrmann et al., 2019; Lewis et al., 2019). Despite this progress, there are still many limitations facing neural text summarization (Kryscinski et al., 2019), the most serious of which is their tendency to generate summaries that are not factually consistent with the input document; a factually consistent summary only contains statements that can be derived from the source document. Recent studies show that about 30% of the summaries generated by neural network sequence-to-sequence models suffer from fact fabrication (Cao et al., 2018). Unfortunately, the widely used ROUGE score is inadequate to quantify factual consistency (Kryscinski et al., 2019). Factual inconsistency can occur at either the entity or the relation level. At the entity level, a model generated summary may contain named- entities that never appeared in the source document. We call this the entity hallucination problem. For example, consider the following model generated summary: People in Italy and the Netherlands are more likely to consume fewer cups of coffee than those in the UK, a study suggests. “UK” never appeared in the input source document (taken from the test set of the XSUM dataset (Narayan et al., 2018)). In fact, the source document mentioned a study involving people in Italy and Netherlands; “UK” was a result of model hallucination. Another type of inconsistency occurs when the entities indeed exist in the source document but the relations between them are not in the source document. This type of inconsistency is much harder to identify. Open Information Extraction (OpenIE) and dependency parsing tools have been used (Cao et al., 2018) to identify the underlying relations in a summary, but are not yet accurate enough for practical use. Ultimately, these researchers relied on manually classifying generated summaries into faithful, fake, or unclear. In this paper, we propose a set of simple metrics to quantify factual consistency at the entity- level. We …

Content truncated to 500 words