TeSum: Human-Generated Abstractive Summarization Corpus for Telugu

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5712–5722 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5712 TeSum: Human-Generated Abstractive Summarization Corpus for Telugu Ashok Urlana∗, Nirmal Surange∗, Pavan Baswani, Priyanka Ravva, Manish Shrivastava Language Technologies Research Center, KCIS, IIIT Hyderabad, India {ashok.urlana,nirmal.surange,pavan.baswani,priyanka.ravva}@research.iiit.ac.in,m.shrivastava@iiit.ac.in Abstract Expert human annotation for summarization is definitely an expensive task, and can not be done on huge scales. But with this work, we show that even with a crowd sourced summary generation approach, quality can be controlled by aggressive expert informed filtering and sampling-based human evaluation. We propose a pipeline that crowd-sources summarization data and then aggressively filters the content via: automatic and partial expert evaluation. Using this pipeline we create a high-quality Telugu Abstractive Summarization dataset (TeSum) which we validate with sampling-based human evaluation. We also provide baseline numbers for various models commonly used for summarization. A number of recently released datasets for summarization, scraped the web-content relying on the assumption that summary is made available with the article by the publishers. While this assumption holds for multiple resources (or news-sites) in English, it should not be generalised across languages without thorough analysis and verification. Our analysis clearly shows that this assumption does not hold true for most Indian language news resources. We show that our proposed filtration pipeline can even be applied to these large-scale scraped datasets to extract better quality article-summary pairs. Keywords: Summarization, Abstractive Summarization, Telugu Dataset, Low Resource Languages 1. Introduction Summarization is a task that has held the interest of the NLP community since the beginning of the computational processing of languages. In NLP literature going back to the early years of NLP, we find a large number of different goals being described as the task of summarization. The DUC (2003-2007) guidelines define summarization as a task of generating a very short text which gives a general idea about the source article. Earlier, Hovy and Lin (1998) asked the question: What is a summary precisely? They proceed to answer the question as : A summary is a text that is produced out of one or more (possibly multimedia) texts, that contains (some of) the same information of the original text(s), and that is no longer than half of the original text(s). Hovy and Lin (1998) and KS Jones (1998) set out to define the task of summarization as something more than a small piece of the text providing an indication of the content of the source article. In fact, Hovy and Lin (1998) follow and extend (Jones, 1998) to provide fine categorization of summary types based on the broader aspects of input, purpose, and output. In recent years, much work has been done to advance the state of the art of summarization for multiple lan- guages across the world. But, most of these works ad- here more closely to the DUC summarization challenge rather than the nuanced definitions presented by Hovy and Lin (1998). We find that collection of …

Content truncated to 500 words