Cross-lingual Fine-tuning for Abstractive Arabic Text Summarization

Proceedings of Recent Advances in Natural Language Processing, pages 655–663 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_074 655 Cross-lingual Fine-tuning for Abstractive Arabic Text Summarization Mram Kahla and Zijian Gy˝oz˝o Yang and Attila Nov´ak P´azm´any P´eter Catholic University Faculty of Information Technology and Bionics Pr´ater u. 50/a, 1083 Budapest, Hungary {lastname.firstname(.midname)}@itk.ppke.hu Abstract While abstractive summarization in certain languages, like English, has already reached fairly good results due to the availability of trend-setting resources, like the CNN/Daily Mail dataset, and considerable progress in generative neural models, progress in abstrac- tive summarization for Arabic, the fifth most- spoken language globally, is still in baby shoes. While some resources for extractive summa- rization have been available for some time, in this paper, we present the first corpus of human-written abstractive news summaries in Arabic, hoping to lay the foundation of this line of research for this important language. The dataset consists of more than 21 thousand items. We used this dataset to train a set of neu- ral abstractive summarization systems for Ara- bic by fine-tuning pre-trained language mod- els such as multilingual BERT, AraBERT, and multilingual BART-50. As the Arabic dataset is much smaller than e.g. the CNN/Daily Mail dataset, we also applied cross-lingual knowledge transfer to significantly improve the performance of our baseline systems. The setups included two M-BERT-based summa- rization models originally trained for Hungar- ian/English and a similar system based on M- BART-50 originally trained for Russian that were further fine-tuned for Arabic. Evalua- tion of the models was performed in terms of ROUGE, and a manual evaluation of flu- ency and adequacy of the models was also per- formed. 1 Introduction When we talk about text summarization that prac- tically means that using certain algorithms, we teach a machine to subtract information from an extensive text and provide a significantly shorter overview of it. And just like in the case of human beings, like a number of cases in our own school experiences, there are two ways of doing that. The first way is called extractive summarization (Nallapati et al., 2016; Zhang et al., 2018; Narayan et al., 2018). In this method the idea is to practi- cally highlight, take out certain keywords, phrases, or sentences from the text and put them together. Therefore, the result, the summary will use the exact same words, terms, and sentences as the orig- inal, and almost certainly even in the same order. It is practically the same method, when we humans glance through a massive text, like a thick script, or a book in just a few minutes. The machine would, just like our mind, focus on the first few words, paragraph, or page, then pick up the most commonly occurring words, and extract complete sentences with them, without changing anything in the sentence. The second way is abstractive summarization (See et al., 2017; Paulus et al., 2017; Rush et al., 2015). Once again, this is not something new for our brain. Like in so many of our school studies, when a certain assignment …

Content truncated to 500 words