DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion

Proceedings of NAACL-HLT 2019, pages 3443–3455 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3443 DISCOFUSE: A Large-Scale Dataset for Discourse-Based Sentence Fusion Mor Geva ∗ Tel Aviv University morgeva@mail.tau.ac.il Eric Malmi Google AI emalmi@google.com Idan Szpektor Google AI szpektor@google.com Jonathan Berant † Tel Aviv University, Allen Institute for AI joberant@cs.tau.ac.il Abstract Sentence fusion is the task of joining several independent sentences into a single coherent text. Current datasets for sentence fusion are small and insufficient for training modern neu- ral models. In this paper, we propose a method for automatically-generating fusion examples from raw text and present DISCOFUSE, a large scale dataset for discourse-based sentence fu- sion. We author a set of rules for identifying a diverse set of discourse phenomena in raw text, and decomposing the text into two in- dependent sentences. We apply our approach on two document collections: Wikipedia and Sports articles, yielding 60 million fusion ex- amples annotated with discourse information required to reconstruct the fused text. We de- velop a sequence-to-sequence model on DIS- COFUSE and thoroughly analyze its strengths and weaknesses with respect to the various dis- course phenomena, using both automatic as well as human evaluation. Finally, we con- duct transfer learning experiments with WEB- SPLIT, a recent dataset for text simplifica- tion. We show that pretraining on DISCOFUSE substantially improves performance on WEB- SPLIT when viewed as a sentence fusion task. 1 Introduction Sentence fusion is the task of combining several independent sentences into a single coherent text (Barzilay and McKeown, 2005). Sentence fusion is important in many NLP applications, including retrieval-based dialogue (Song et al., 2018; Yan and Zhao, 2018), text summarization (Barzilay and McKeown, 2005; Bing et al., 2015) and ques- tion answering (Li et al., 2018; Marsi and Krah- mer, 2005). Such systems retrieve multiple sen- tences from different sources, documents or para- graphs, and use them to construct a coherent text. ∗Work done during internship at Google AI. † Work done at Google AI. Figure 1: Example for two independent sentences, and their fusion. The modifications applied are pronomi- nalization (blue) and connective insertion (red). Sentence fusion is challenging because it re- quires understanding the discourse semantics be- tween the input sentences. Consider the example in Figure 1: a coherent fusion of the sentences re- quires understanding that the second sentence con- trasts the first one, in order to insert the discourse connective “However”. In addition, the gender and syntactic role of the entity “Zeitler” needs to be inferred to insert the pronoun “he”. Prior work on sentence fusion (Barzilay and McKeown, 2005; Turner and Charniak, 2005; Filippova, 2010; Elsner and Santhanam, 2011; Thadani and McKeown, 2013; Bing et al., 2015; Chali et al., 2017) utilized very small amounts of labeled data, which are insufficient to train modern neural models. In this work, we propose a method for automatically generating sentence fusion ex- amples at scale from raw text corpora. To this end, we go over sentences and contigu- ous pairs of …

Content truncated to 500 words