Evaluating the Factual Consistency of Abstractive Text Summarization

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9332–9346, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9332 Evaluating the Factual Consistency of Abstractive Text Summarization Wojciech Kry´sci´nski, Bryan McCann, Caiming Xiong, Richard Socher Salesforce Research {kryscinski,bmccann,cxiong,rsocher}@salesforce.com Abstract The most common metrics for assessing summarization algorithms do not account for whether summaries are factually consis- tent with source documents. We propose a weakly-supervised, model-based approach for verifying factual consistency and iden- tifying conflicts between source documents and generated summaries. Training data is generated by applying a series of rule-based transformations to the sentences of source documents. The factual consistency model is then trained jointly for three tasks: 1) predict whether each summary sentence is factually consistent or not, 2) in either case, extract a span in the source document to support this consistency prediction, 3) for each summary sentence that is deemed inconsistent, extract the inconsistent span from it. Transferring this model to summaries generated by several neural models reveals that this highly scalable approach outperforms previous models, including those trained with strong super- vision using datasets from related domains, such as natural language inference and fact checking. Additionally, human evaluation shows that the auxiliary span extraction tasks provide useful assistance in the process of verifying factual consistency. We also release a manually annotated dataset for factual consistency verification, code for training data generation, and trained model weights at https://github.com/salesforce/factCC. 1 Introduction The goal of text summarization is to transduce long documents into a shorter form that retains the most important aspects from the source doc- ument. Common approaches to summarization are extractive (Dorr et al., 2003; Nallapati et al., 2017) where models directly copy salient parts of the source document into the summary, abstrac- tive (Rush et al., 2015; Paulus et al., 2017) where the important parts are paraphrased to form novel sentences, and hybrid (Gehrmann et al., 2018), combining the two methods by employing special- ized extractive and abstractive components. Advancements in neural architec- tures (Cho et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017), transfer learn- ing (McCann et al., 2017; Devlin et al., 2018), and availability of large-scale supervised datasets (Nallapati et al., 2016; Grusky et al., 2018) allowed deep learning-based approaches to dominate the field. State-of-the-art solutions uti- lize self-attentive Transformer blocks (Liu, 2019; Liu and Lapata, 2019), attention and copying mechanisms (See et al., 2017; Cohan et al., 2018), and multi-objective training strategies (Guo et al., 2018; Pasunuru and Bansal, 2018), including rein- forcement learning techniques (Kry´sci´nski et al., 2018; Dong et al., 2018; Wu and Hu, 2018). Despite significant efforts made by the re- search community, there are still many challenges limiting progress in summarization: insufficient evaluation protocols that omit important dimen- sions, such as factual consistency, noisy datasets that leave the task underconstrained, and strong, domain-specific layout biases in the data that dom- inate training signal (Kry´sci´nski et al., 2019). We address the problem of verifying factual consistency between source documents and gener- ated summaries: …

Content truncated to 500 words