WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4034–4048 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4034 WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization Faisal Ladhak1∗, Esin Durmus2∗, Claire Cardie2, and Kathleen McKeown1 1Columbia University, New York, NY 2Cornell University, Ithaca, NY {faisal,kathy}@cs.columbia.edu {ed459}@cornell.edu, {cardie}@cs.cornell.edu Abstract We introduce WikiLingua, a large-scale, mul- tilingual dataset for the evaluation of cross- lingual abstractive summarization systems. We extract article and summary pairs in 18 languages from WikiHow12, a high quality, collaborative resource of how-to guides on a diverse set of topics written by human authors. We create gold-standard article- summary alignments across languages by aligning the images that are used to describe each how-to step in an article. As a set of baselines for further studies, we evaluate the performance of existing cross-lingual abstrac- tive summarization methods on our dataset. We further propose a method for direct cross- lingual summarization (i.e., without requiring translation at inference time) by leveraging synthetic data and Neural Machine Transla- tion as a pre-training step. Our method signif- icantly outperforms the baseline approaches, while being more cost efficient during infer- ence. 1 Introduction Although there has been a tremendous amount of progress in abstractive summarization in recent years, most research has focused on monolingual summarization because of the lack of high quality multilingual resources (Lewis et al., 2019a; Song et al., 2020). While there have been a few studies to address the lack of resources for cross-lingual summarization (Giannakopoulos, 2013; Li et al., 2013; Elhadad et al., 2013; Nguyen and Daum´e III, 2019), the datasets employed are very limited in size. Scarcity in the availability of data for cross- lingual abstractive summarization can largely be at- tributed to the difficulty of collecting high-quality, ∗Equal contribution. 1https://www.wikihow.com 2The data was collected in accordance with the terms and conditions listed on the website. large-scale datasets via crowd-sourcing. It is a costly endeavor, since it requires humans to read, comprehend, condense, and paraphrase entire ar- ticles. Moreover, subjectivity in content selection, i.e. identifying the salient points of a given article, only adds to the difficulty of crowd sourcing this task (Nguyen and Daum´e III, 2019). To overcome the lack of a large-scale, high qual- ity resource for cross-lingual summarization, we present a new benchmark dataset, WikiLingua,3 which consists of collaboratively written how-to guides with gold-standard summaries across 18 lan- guages. Each article and summary is written and edited by 23 people, and further reviewed by 16 people, on average, which ensures that the content is of a high-quality. The articles describe multi- ple methods with steps to complete a procedural task from a diverse set of topics, such as “How to Make a Creamy Coffee”, “How to Exercise to Ease Back Pain”. Each step contains a one sen- tence summary followed by a paragraph detailing the instruction, along with an image to illustrate the given instruction, as shown in Figure 1. Since the ordering of steps may differ for the same article across languages, …

Content truncated to 500 words