Ensure the Correctness of the Summary: Incorporate Entailment Knowledge into Abstractive Sentence Summarization

Proceedings of the 27th International Conference on Computational Linguistics, pages 1430–1441 Santa Fe, New Mexico, USA, August 20-26, 2018. 1430 Ensure the Correctness of the Summary: Incorporate Entailment Knowledge into Abstractive Sentence Summarization Haoran Li1,2, Junnan Zhu1,2, Jiajun Zhang1,2 and Chengqing Zong1,2,3 1 National Laboratory of Pattern Recognition, CASIA, Beijing, China 2 University of Chinese Academy of Sciences, Beijing, China 3 CAS Center for Excellence in Brain Science and Intelligence Technology {haoran.li, junnan.zhu, jjzhang, cqzong}@nlpr.ia.ac.cn Abstract In this paper, we investigate the sentence summarization task that produces a summary from a source sentence. Neural sequence-to-sequence models have gained considerable success for this task, while most existing approaches only focus on improving word overlap between the generated summary and the reference, which ignore the correctness, i.e., the summary should not contain error messages with respect to the source sentence. We argue that correctness is an essential requirement for summarization systems. Considering a correct summary is seman- tically entailed by the source sentence, we incorporate entailment knowledge into abstractive summarization models. We propose an entailment-aware encoder under multi-task framework (i.e., summarization generation and entailment recognition) and an entailment-aware decoder by entailment Reward Augmented Maximum Likelihood (RAML) training. Experimental results demonstrate that our models significantly outperform baselines from the aspects of informative- ness and correctness. 1 Introduction Sentence summarization is a well-studied task that creates a condensed version of a long source sentence. Sequence-to-sequence (seq2seq) model that encodes a source sequence into a latent representation and outputs another sequence is the dominating framework for sentence summarization (Rush et al., 2015; Chopra et al., 2016; Takase et al., 2016; Zhou et al., 2017; Li et al., 2017b; Li et al., 2018). Despite substantial improvements on this task, most of the existing researches typically aim to improve word overlap between the generated summary and the references, which is measured by n-gram matching metrics (e.g., ROUGE (Lin, 2004)). Hence, it cannot guarantee the semantic correctness of the summary as a whole. Therefore, in some cases, the summary giving high matching scores may contain critical error messages, which makes the summary fail to capture the correct information with respect to the source sentence. Previous study shows that about 30% of the summaries generated by state-of-the-art seq2seq system are subject to this problem (Cao et al., 2017). Here is an example (the digits are replaced by “#”): Source sentence: franch won the gold medal at women ’s epee team event of the fie #### world championships by beating china ##-## . Reference: france beats china for women ’s epee team gold State-of-the-art seq2seq model: canada wins women ’s epee team event For the example shown above, the seq2seq system produces a fluent summary which contains an ob- vious mistake. The true winner of the “women ’s epee team event” is “france”, while the summarization model wrongly generates “canada”, which is probably due to similar word representations for country names. Though the word overlap between the generated summary and the reference is considerable, leading to high ROUGE scores, the …

Content truncated to 500 words