Small but Mighty: New Benchmarks for Split and Rephrase

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1198–1205, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1198 Small but Mighty: New Benchmarks for Split and Rephrase Li Zhang♠∗ Huaiyu Zhu♦ ♠University of Pennsylvania ♦IBM Research ♣ Google Research zharry@seas.upenn.edu {huaiyu,yunyaoli}@us.ibm.com sidbrahma@google.com Siddhartha Brahma♣† Yunyao Li♦ Abstract Split and Rephrase is a text simplification task of rewriting a complex sentence into sim- pler ones. As a relatively new task, it is paramount to ensure the soundness of its eval- uation benchmark and metric. We find that the widely used benchmark dataset univer- sally contains easily exploitable syntactic cues caused by its automatic generation process. Taking advantage of such cues, we show that even a simple rule-based model can perform on par with the state-of-the-art model. To rem- edy such limitations, we collect and release two crowdsourced benchmark datasets. We not only make sure that they contain signifi- cantly more diverse syntax, but also carefully control for their quality according to a well- defined set of criteria. While no satisfactory automatic metric exists, we apply fine-grained manual evaluation based on these criteria us- ing crowdsourcing, showing that our datasets better represent the task and are significantly more challenging for the models.1 1 Introduction Split and Rephrase is the task of rewriting a pre- sumably long and complex sentence into shorter and simpler sentences, while maintaining the same meaning. For example, one possible way to split the sentence “Voiced by Aoi Koga, Kaguya is the series’ titular character, popular among a wide au- dience.” would result in “Kaguya is voiced by Aoi Koga. Kaguya is the series’ titular charac- ter. Kaguya is popular among a wide audience.” While the split sentences have to be coherent, para- phrasing is not enforced. For example, the word “titular” does not have to be replaced. This type of ∗ Work done during internship at IBM Research. † Work done during employment at IBM Research. 1Our datasets and code will be available at https:// github.com/System-T/TextSimplification. text simplification is challenging as its natural lan- guage generation process potentially involves mul- tiple sub-processes such as co-reference resolution, named-entity recognition, semantic role labelling, etc. Split and Rephrase has two main real-world uses: first, to benefit systems whose performance improves with decreasing length of sentences e.g. entity extraction (Zhang et al., 2017) and machine translation (Koehn and Knowles, 2017) by acting as a pre-processing step; second, to benefit human readers, especially those less proficient with the lan- guage when reading complex documents such as terms and agreements, in understanding the mean- ing more easily and accurately (Inui et al., 2003; Siddharthan, 2002). Datasets of the Split and Rephrase task con- tain pairs of a complex sentence and a presum- ably meaning-preserving simplified rewrite con- taining multiple simpler sentences. The task was introduced by Narayan et al. (2017), with the re- lease of the WebSplit corpus. Afterwards, Aharoni and Goldberg (2018) proposed the state-of-the-art model to date, a sequence-to-sequence model (Bah- danau et al., 2015) with a …

Content truncated to 500 words