Understanding the Behaviour of Neural Abstractive Summarizers using Contrastive Examples

Proceedings of NAACL-HLT 2019, pages 3949–3954 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3949 Understanding the Behaviour of Neural Abstractive Summarizers using Contrastive Examples Krtin Kumar School of Computer Science McGill University krtin.kumar@mail.mcgill.ca Jackie Chi Kit Cheung School of Computer Science McGill University jcheung@cs.mcgill.ca Abstract Neural abstractive summarizers generate sum- mary texts using a language model condi- tioned on the input source text, and have recently achieved high ROUGE scores on benchmark summarization datasets. We inves- tigate how they achieve this performance with respect to human-written gold-standard ab- stracts, and whether the systems are able to un- derstand deeper syntactic and semantic struc- tures. We generate a set of contrastive sum- maries which are perturbed, deficient versions of human-written summaries, and test whether existing neural summarizers score them more highly than the human-written summaries. We analyze their performance on different datasets and find that these systems fail to understand the source text, in a majority of the cases. 1 Introduction Open-domain abstractive summarization is a long- standing goal of the field of automatic summariza- tion. Compared to extractive techniques, abstrac- tion offers the potential to generate much more useful summaries by simplifying and rephrasing the source text (Knight and Marcu, 2002), and furthermore by aggregating information and per- forming operations which are not possible with extractive techniques (Genest and Lapalme, 2012; Carenini and Cheung, 2008). Recently, a number of abstractive summa- rization systems based on neural sequence-to- sequence architectures have been proposed (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017; Paulus et al., 2018; Chen and Bansal, 2018). These systems learn a compressed representation of the source text using an encoder, then generate the output summary using a conditional decoder. Such neural abstractive systems have achieved very good ROUGE scores on different datasets. Source A former Iraqi army chief of staff being investigated in denmark for war crimes is believed to be back in Iraq, one of his sons said Tuesday. Contrastive 1 Iraqi general missing in denmark believed to be back from Iraq. Contrastive 2 Iraqi general missing from Iraq believed to be back in Iraq. Table 1: Examples of generated contrastive summaries. Bold indicate switched words . Our interest in this paper is to investigate how these abstractive systems achieve such results, and whether they represent progress towards language understanding and generation. ROUGE arguably provides a limited view of the performance of such systems, as they only relate the system summary to a fixed number of gold-standard summaries. We propose a novel method to directly test the abstrac- tive summarizers in terms of how they score po- tential candidate summaries, viewing them as con- ditional language models. This allows us to test whether the summarizers favour output summaries with specific desired qualities, such as generating a summary that is semantically consistent and en- tailed by the source text. We test how well the neural abstractive sum- marizers distinguish human-written abstracts from contrastive distractors, which are clearly incorrect summaries …

Content truncated to 500 words