Learning to Fuse Sentences with Transformers for Summarization

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4136–4142, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4136 Learning to Fuse Sentences with Transformers for Summarization Logan Lebanoff† Franck Dernoncourt§ Doo Soon Kim§ Lidan Wang§ Walter Chang§ Fei Liu† †University of Central Florida §Adobe Research loganlebanoff@knights.ucf.edu feiliu@cs.ucf.edu {dernonco,dkim,lidwang,wachang}@adobe.com Abstract The ability to fuse sentences is highly attrac- tive for summarization systems because it is an essential step to produce succinct abstracts. However, to date, summarizers can fail on fus- ing sentences. They tend to produce few sum- mary sentences by fusion or generate incorrect fusions that lead the summary to fail to retain the original meaning. In this paper, we explore the ability of Transformers to fuse sentences and propose novel algorithms to enhance their ability to perform sentence fusion by leverag- ing the knowledge of points of correspondence between sentences. Through extensive experi- ments, we investigate the effects of different design choices on Transformer’s performance. Our findings highlight the importance of mod- eling points of correspondence between sen- tences for effective sentence fusion. 1 Introduction A renewed emphasis must be placed on sentence fu- sion in the context of neural abstractive summariza- tion. A majority of the systems are trained end-to- end (See et al., 2017; Paulus et al., 2018; Narayan et al., 2018; Chen and Bansal, 2018; Gehrmann et al., 2018; Liu and Lapata, 2019), where an ab- stractive summarizer is rewarded for generating summaries that contain the same words as human abstracts, measured by automatic metrics such as ROUGE (Lin, 2004). A summarizer, however, is not rewarded for correctly fusing sentences. In fact, when examined more closely, only few sen- tences in system abstracts are generated by fu- sion (Falke et al., 2019; Lebanoff et al., 2019). For instance, 6% of summary sentences generated by Pointer-Gen (See et al., 2017) are through fusion, whereas human abstracts contain 32% fusion sen- tences. Moreover, sentences generated by fusion are prone to errors. They can be ungrammatical, nonsensical, or otherwise ill-formed. There is thus an urgent need to develop neural abstractive sum- marizers to fuse sentences properly. The importance of sentence fusion has long been recognized by the community before the era of neu- ral text summarization. The pioneering work of Barzilay et al. (1999) introduces an information fusion algorithm that combines similar elements across related text to generate a succinct summary. Later work, such as (Marsi and Krahmer, 2005; Filippova and Strube, 2008; Elsner and Santhanam, 2011; Thadani and McKeown, 2013; Mehdad et al., 2013), builds a dependency or word graph by com- bining syntactic trees of similar sentences, then employs integer linear programming to decode a summary sentence from the graph. Most of these studies have assumed a set of similar sentences as input, where fusion is necessary to reduce repeti- tion. Nonetheless, humans do not limit themselves to combine similar sentences. In this paper, we pay particular attention to fuse disparate sentences that contain fundamentally different content but remain related to make …

Content truncated to 500 words