Learn to Copy from the Copying History: Correlational Copy Network for Abstractive Summarization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4091–4101 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4091 Learn to Copy from the Copying History: Correlational Copy Network for Abstractive Summarization Haoran Li1, Song Xu1, Peng Yuan1, Yujia Wang2, Youzheng Wu1, Xiaodong He1, Bowen Zhou1 1 JD AI Research 2 University of California, Berkeley {lihaoran24, xusong28, yuanpeng29}@jd.com Abstract The copying mechanism has had considerable success in abstractive summarization, facili- tating models to directly copy words from the input text to the output summary. Exist- ing works mostly employ encoder-decoder at- tention, which applies copying at each time step independently of the former ones. How- ever, this may sometimes lead to incomplete copying. In this paper, we propose a novel copying scheme named Correlational Copying Network (CoCoNet) that enhances the stan- dard copying mechanism by keeping track of the copying history. It thereby takes ad- vantage of prior copying distributions and, at each time step, explicitly encourages the model to copy the input word that is rele- vant to the previously copied one. In ad- dition, we strengthen CoCoNet through pre- training with suitable corpora that simulate the copying behaviors. Experimental results show that CoCoNet can copy more accu- rately and achieves new state-of-the-art per- formances on summarization benchmarks, in- cluding CNN/DailyMail for news summariza- tion and SAMSum for dialogue summariza- tion. Our code is available at https:// github.com/hrlinlp/coconet. 1 Introduction Text summarization techniques (Rush et al., 2015; Chopra et al., 2016; Zhou et al., 2017; Li et al., 2018; Zhang et al., 2018; Li et al., 2019, 2020a,b; Xu et al., 2020a; Yuan et al., 2020) aim to gener- ate a condensed and cohesive version of the input text, enabling readers to grasp the main points with- out reading the full text. There are two types of summarizers: extractive and abstractive. Extractive methods produce a summary by taking important sentences from the original text and combining these extracts, while abstractive methods involve interpreting and paraphrasing the input when gen- erating a summary. The latter is more similar to Dialogue Ernest: hey Mike , did you park your car on our street? Mike: no, took it into garage today Ernest: ok good Mike: why? Ernest: someone just crashed into a red Hon- da looking just like yours Mike: lol lucky me Summary Mike took his car into garage today. Ernest is relieved as someone had just crashed into a red Honda which looks like Mike’s. Table 1: An example from the dialogue summariza- tion task. Highlighted words are copied consecutively from the input. Previously copied words (such as “just crashed”) can guide the following copying operations (such as“into a red Honda”). how humans would summarize a text, but it is far more challenging to achieve. Currently, the sequence-to-sequence (Seq2Seq) framework has become the mainstream for per- forming abstractive summarization tasks. However, it suffers from handling out-of-vocabulary words (OOV). As it has been observed that some words in the input text reappear in the summary, one way …

Content truncated to 500 words