Reducing Quantity Hallucinations in Abstractive Summarization

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2237–2249 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2237 Reducing Quantity Hallucinations in Abstractive Summarization Zheng Zhao Shay B. Cohen Bonnie Webber Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB zheng.zhao@ed.ac.uk , {scohen,bonnie}@inf.ed.ac.uk Abstract It is well-known that abstractive summaries are subject to hallucination—including mate- rial that is not supported by the original text. While summaries can be made hallucination- free by limiting them to general phrases, such summaries would fail to be very informative. Alternatively, one can try to avoid hallucina- tions by verifying that any specific entities in the summary appear in the original text in a similar context. This is the approach taken by our system, HERMAN. The system learns to recognize and verify quantity entities (dates, numbers, sums of money, etc.) in a beam- worth of abstractive summaries produced by state-of-the-art models, in order to up-rank those summaries whose quantity terms are sup- ported by the original text. Experimental re- sults demonstrate that the ROUGE scores of such up-ranked summaries have a higher Pre- cision than summaries that have not been up- ranked, without a comparable loss in Recall, resulting in higher F1. Preliminary human evaluation of up-ranked vs. original sum- maries shows people’s preference for the for- mer. 1 Introduction Automatic summarization is the task of compress- ing a lengthy text to a more concise version that preserves the information of the original text. Com- mon approaches are either extractive, selecting and assembling salient words, phrases and sentences from the source text to form the summary (Lin and Bilmes, 2011; Nallapati et al., 2017; Narayan et al., 2018b), or abstractive, generating the summary from scratch, containing novel words and phrases that are paraphrased from important parts of the original text (Clarke and Lapata, 2008; Rush et al., 2015; Wang et al., 2019). The latter is more chal- lenging as it involves human-like capabilities, e.g., paraphrasing, generalizing, inferring and including Article: . .. the volcano was still spewing ash on Sunday, hampering rescue operations. More than a dozen people were killed when it erupted in 2014 ...rescue teams are still scouring the area, looking for more victims who may have been killed or badly burned . . . Summary: Rescue teams in Indonesia are searching for more than 20 people missing after the Mount Sinabung volcano erupted on Saturday, killing at least 11 people and injuring at least 20 others. Article: The scale of the criminal operation has been de- tailed by the three sources, who say they were . . . a victim of the fraud shown the call centre script has confirmed it matched the one read out to her when she was conned out of £5,000 . . . Summary: Three whistleblowers have told the BBC that they were involved in a scam that conned hundreds of TalkTalk customers out of more than £100,000. Article: The government and the doctors’ …

Content truncated to 500 words