Annotating and Modeling Fine-grained Factuality in Summarization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1449–1462 June 6–11, 2021. ©2021 Association for Computational Linguistics 1449 Annotating and Modeling Fine-grained Factuality in Summarization Tanya Goyal and Greg Durrett Department of Computer Science The University of Texas at Austin tanyagoyal@utexas.edu, gdurrett@cs.utexas.edu Abstract Recent pre-trained abstractive summarization systems have started to achieve credible per- formance, but a major barrier to their use in practice is their propensity to output sum- maries that are not faithful to the input and that contain factual errors. While a number of annotated datasets and statistical models for assessing factuality have been explored, there is no clear picture of what errors are most important to target or where current tech- niques are succeeding and failing. We ex- plore both synthetic and human-labeled data sources for training models to identify factual errors in summarization, and study factuality at the word-, dependency-, and sentence-level. Our observations are threefold. First, exhib- ited factual errors differ significantly across datasets, and commonly-used training sets of simple synthetic errors do not reflect errors made on abstractive datasets like XSUM. Sec- ond, human-labeled data with fine-grained an- notations provides a more effective training signal than sentence-level annotations or syn- thetic data. Finally, we show that our best factuality detection model enables training of more factual XSUM summarization models by allowing us to identify non-factual tokens in the training data.1 1 Introduction Hallucination of unsupported or incorrect facts is a known shortcoming of current text genera- tion and summarization models (Cao et al., 2018; Falke et al., 2019). This has been established for both abstractive summarization models (Maynez et al., 2020) and extractive summarization models (Kryscinski et al., 2020; Falke et al., 2019). Past work has explored using off-the-shelf frameworks such as entailment models (Falke et al., 2019) or QA systems (Durmus et al., 2020; Wang et al., 1Code and data available at https://github.com/ tagoyal/factuality-datasets 2020) to detect and sometimes correct errors in generated summaries. Another line of recent work has used synthetically generated data to specifi- cally train models on the factuality detection task (Kryscinski et al., 2020; Zhao et al., 2020; Goyal and Durrett, 2020a). However, these efforts have focused on different datasets, summarization sys- tems, and error types, often shedding little light on what errors state-of-the-art systems are actually making and how to fix them. In this paper, we aim to answer two main ques- tions. First, while synthetic data generation ap- proaches are specifically designed for factuality evaluation, do these align with actual errors made by generation models? We find the an- swer is no: techniques using surface-level data cor- ruption (Kryscinski et al., 2020; Zhao et al., 2020; Cao et al., 2020) or paraphrasing (Goyal and Dur- rett, 2020a) target inherently different error distri- butions than those seen in actual model generations, and factuality models trained on these datasets per- form poorly in practice. Furthermore, we show that different summarization domains, CNN/Daily Mail (Hermann et al., …

Content truncated to 500 words