Improving Truthfulness of Headline Generation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1335–1346 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1335 Improving Truthfulness of Headline Generation Kazuki Matsumaru Sho Takase Tokyo Institute of Technology {kazuki.matsumaru,sho.takase,naoaki.okazaki}@nlp.c.titech.ac.jp Naoaki Okazaki Abstract Most studies on abstractive summarization re- port ROUGE scores between system and ref- erence summaries. However, we have a con- cern about the truthfulness of generated sum- maries: whether all facts of a generated sum- mary are mentioned in the source text. This paper explores improving the truthfulness in headline generation on two popular datasets. Analyzing headlines generated by the state- of-the-art encoder-decoder model, we show that the model sometimes generates untruthful headlines. We conjecture that one of the rea- sons lies in untruthful supervision data used for training the model. In order to quantify the truthfulness of article-headline pairs, we consider the textual entailment of whether an article entails its headline. After confirming quite a few untruthful instances in the datasets, this study hypothesizes that removing untruth- ful instances from the supervision data may remedy the problem of the untruthful behav- iors of the model. Building a binary classifier that predicts an entailment relation between an article and its headline, we filter out untruth- ful instances from the supervision data. Exper- imental results demonstrate that the headline generation model trained on filtered supervi- sion data shows no clear difference in ROUGE scores but remarkable improvements in auto- matic and manual evaluations of the generated headlines. 1 Introduction Automatic text summarization aims at condensing a text into a shorter version while maintaining the essential information (Mani, 2001). Methods on summarization are broadly categorized into two approaches: extractive and abstractive. The former extracts important words, phrases, or sentences from a source text to compile a summary (Gold- stein et al., 2000; Erkan and Radev, 2004; Mihalcea, 2004; Lin and Bilmes, 2011). In contrast, the latter involves more complex linguistic operations (e.g., abstraction, paraphrasing, and compression) to gen- erate a new text (Knight and Marcu, 2000; Clarke and Lapata, 2008). Until 2014, abstractive summa- rization had been less popular than extractive one because of the difficulty of generating a natural text. However, research on abstractive summarization has attracted a lot of attentions recently with the advances on encoder-decoder models (Rush et al., 2015; Takase et al., 2016; Zhou et al., 2017; Cao et al., 2018a; Song et al., 2019; Wang et al., 2019). English Gigaword (Graff and Cieri, 2003; Napoles et al., 2012) is a representative dataset for abstractive summarization. Rush et al. (2015) regarded Gigaword as a corpus containing a large number of article-headline pairs for training an encoder-decoder model. Their work assumed a task setting where the first sentence of an article is a source text and its corresponding headline is a target text (summary). Since then, it has been a common practice to use the Gigaword dataset with this task setting and to measure the quality of generated headlines with ROUGE scores (Lin and Hovy, 2003) between …

Content truncated to 500 words