ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9391–9407 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9391 ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization Alireza Salemi1, Emad Kebriaei1, Ghazal Neisi Minaei1, Azadeh Shakery1,2 1School of Electrical and Computer Engineering College of Engineering, University of Tehran, Tehran, Iran 2School of Computer Science Institute for Research in Fundamental Sciences (IPM), Iran {alireza.salemi,emad.kebriaei,ghazal.minaei,shakery}@ut.ac.ir Abstract Abstractive text summarization is one of the ar- eas influenced by the emergence of pre-trained language models. Current pre-training works in abstractive summarization give more points to the summaries with more words in com- mon with the main text and pay less attention to the semantic similarity between generated sentences and the original document. We pro- pose ARMAN, a Transformer-based encoder- decoder model pre-trained with three novel ob- jectives to address this issue. In ARMAN, salient sentences from a document are selected according to a modified semantic score to be masked and form a pseudo summary. To sum- marize more accurately and similar to human writing patterns, we applied modified sentence reordering. We evaluated our proposed mod- els on six downstream Persian summarization tasks. Experimental results show that our pro- posed model achieves state-of-the-art perfor- mance on all six summarization tasks mea- sured by ROUGE and BERTScore. Our mod- els also outperform prior works in textual en- tailment, question paraphrasing, and multiple choice question answering. Finally, we estab- lished a human evaluation and show that using the semantic score significantly improves sum- marization results. 1 Introduction Abstractive text summarization is the task of gener- ating a short, fluent, and concise text that contains novel words and phrases other than the original document, preserving the primary subjects in the document. In contrast with extractive summariza- tion, which aims to select the most important parts of the text to generate a summary, in abstractive summarization, the main goal is to generate a new persuasive piece of text as the summary of a docu- ment. Earlier abstractive summarization works (Her- mann et al., 2015; See et al., 2017; Rush et al., 2015) focused on training with large datasets con- taining pairs of documents and summaries in a supervised manner. By introducing Transformer (Vaswani et al., 2017) architecture and pre-training objectives and their positive impact on most NLP tasks, most current state-of-the-art (SOTA) meth- ods focused on self-supervised objectives for pre- training Transformer architecture in abstractive summarization tasks (Liu and Lapata, 2019; Zhang et al., 2020a; Qi et al., 2020). However, current pre-training works give more points to the sum- mary with more words in common with the main text and pay less attention to the semantic similar- ity between generated sentences and the original document. According to Simons (2017), the Persian lan- guage is one of the top 25 spoken languages in the world. However, there are limited research studies in Persian document summarization, and most of the prior works were mainly focused on extractive summarization. The main focus of …

Content truncated to 500 words