Self-Supervised Learning for Contextualized Extractive Summarization

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2221–2227 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2221 Self-Supervised Learning for Contextualized Extractive Summarization Hong Wang†, Xin Wang†, Wenhan Xiong†, Mo Yu‡, Xiaoxiao Guo‡, Shiyu Chang‡, William Yang Wang† † University of California, Santa Barbara ‡ IBM Research {hongwang600, xwang, xwhan, william}@cs.ucsb.edu, yum@us.ibm.com, {xiaoxiao.guo, shiyu.chang}@ibm.com Abstract Existing models for extractive summarization are usually trained from scratch with a cross- entropy loss, which does not explicitly cap- ture the global context at the document level. In this paper, we aim to improve this task by introducing three auxiliary pre-training tasks that learn to capture the document-level con- text in a self-supervised fashion. Experiments on the widely-used CNN/DM dataset vali- date the effectiveness of the proposed auxiliary tasks. Furthermore, we show that after pre- training, a clean model with simple building blocks is able to outperform previous state-of- the-art that are carefully designed. 1 1 Introduction Extractive summarization aims at shortening the original article while retaining the key information through the way of selection sentences from the original articles. This paradigm has been proven effective by many previous systems (Carbonell and Goldstein, 1998; Mihalcea and Tarau, 2004; McDonald, 2007; Cao et al., 2015). In order to decide whether to choose a particular sentence, the system should have a global view of the doc- ument context, e.g., the subject and structure of the document. However, previous works (Nallap- ati et al., 2017; Al-Sabahi et al., 2018; Zhou et al., 2018; Zhang et al., 2018) usually directly build an end-to-end training system to learn to choose sen- tences without explicitly modeling the document context, counting on that the system can automat- ically learn the document-level context. We argue that it is hard for these end-to-end systems to learn to leverage the document context from scratch due to the challenges of this task, and a well pre-trained embedding model that in- corporates document context should help on this 1Code can be found in this repository: https:// github.com/hongwang600/Summarization Last week, I went to attend a one-day meeting. I booked the flight in advanced. [masked sentence] The earliest next flight will be a few days later. I had to use the online discussion instead. But the flight was cancelled due to the weather. But I lost my passport. The meeting was cancelled. The weather is good today. Masked Paragraph Candidate Sentences Figure 1: An example for the Mask pre-training task. A sentence is masked in the original paragraph, and the model is required to predicted the missing sentence from the candidate sentences. task. In recent years, extensive works (Penning- ton et al., 2014; Nie and Bansal, 2017; Lin et al., 2017; Peters et al., 2018; Devlin et al., 2018; Sub- ramanian et al., 2018; Cer et al., 2018; Logeswaran and Lee, 2018; Pagliardini et al., 2018) have been done in learning the word or sentence represen- tations, but most of them only use a sentence or a …

Content truncated to 500 words