Semantically Driven Sentence Fusion: Modeling and Evaluation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1491–1505 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1491 Semantically Driven Sentence Fusion: Modeling and Evaluation Eyal Ben-David1, Orgad Keller2, Eric Malmi2, Idan Szpektor2, Roi Reichart1 1Faculty of Industrial Engineering and Management, Technion, IIT 2Google Research eyalbd12@campus.technion.ac.il, roiri@technion.ac.il {orgad,emalmi,szpektor}@google.com Abstract Sentence fusion is the task of joining related sentences into coherent text. Current training and evaluation schemes for this task are based on single reference ground-truths and do not account for valid fusion variants. We show that this hinders models from robustly captur- ing the semantic relationship between input sentences. To alleviate this, we present an ap- proach in which ground-truth solutions are au- tomatically expanded into multiple references via curated equivalence classes of connective phrases. We apply this method to a large-scale dataset and use the augmented dataset for both model training and evaluation. To improve the learning of semantic representation using multiple references, we enrich the model with auxiliary discourse classification tasks under a multi-tasking framework. Our experiments highlight the improvements of our approach over state-of-the-art models. 1 1 Introduction Generative NLP tasks, such as machine translation and summarization, often rely on human generated ground truth. Datasets for such tasks typically con- tain only a single reference per example. This may result from the costly effort of human annotations, or from collection methodologies that are restricted to single reference resources (e.g., utilizing existing corpora; Koehn, 2005; Nallapati et al., 2016). How- ever, typically there are other possible generation results, such as ground-truth paraphrases, that are also valid. Failing to consider multiple references hurts the development of generative models, since such models are considered correct, at both training and evaluation, only if they follow one specific and often arbitrary generation path per example. In this work we address Sentence Fusion, a chal- lenging task where a model should combine related 1Our code is at https://github.com/eyalbd2/ Semantically-Driven-Sentence-Fusion. sentences, which may overlap in content, into a compact coherent text. The output should preserve the information in the input sentences as well as their semantic relationship. It is a crucial compo- nent in many NLP applications, including text sum- marization, question answering and retrieval-based dialogues (Jing and McKeown, 2000; Barzilay and McKeown, 2005; Marsi and Krahmer, 2005; Lebanoff et al., 2019; Szpektor et al., 2020). Our analysis of state-of-the-art fusion models (Geva et al., 2019; Rothe et al., 2019) indicates that they still struggle to correctly detect the semantic relationship between the input sentences, which is reflected in inappropriate discourse marker se- lection in the generated fusions (§4). At the same time, DISCOFUSE (Geva et al., 2019), the dataset they use, is limited to a single reference per exam- ple, ignoring discourse marker synonyms such as ‘but’ and ‘however’. Noticing that humans tend to judge these synonyms as equally suitable (§3), we hypothesize that relying on single references may limit the performance of those models. To overcome this limitation, we explore an ap- proach in which ground-truth solutions are …

Content truncated to 500 words