Monolingual versus multilingual BERTology for Vietnamese extractive multi-document summarization

Monolingual versus Multilingual BERTology for Vietnamese Extractive Multi-Document Summarization Huy Quoc To, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen, Anh Gia-Tuan Nguyen University of Information Technology, Ho Chi Minh City, Vietnam Vietnam National University, Ho Chi Minh City, Vietnam {huytq,kietnv,ngannlt,anhngt}@uit.edu.vn Abstract Recent researches have demonstrated that BERT shows potential in a wide range of natural language processing tasks. is adopted as an encoder for many state-It of-the-art automatic summarizing sys- tems, which achieve excellent performance. However, so far, there is not much work done for Vietnamese. In this paper, we showcase how BERT can be implemented for extractive text summarization in Viet- namese on multi-document. We intro- duce a novel comparison between different multilingual and monolingual BERT mod- els. The experiment results indicate that monolingual models produce promising re- sults compared to other multilingual mod- els and previous text summarizing models for Vietnamese. 1 Introduction Automatic text summarization is one of the challenging tasks of natural language processing (NLP). This task requires the machine to gen- erate a piece of text which is a shorter version of one or many input documents. The output of the automatic text summarization tool has to be condensed into information while preserving the primary knowledge of the given documents. The applications for this task are widely imple- mented. A simple example is an automatic sum- marizing tool that can generate abstracts for sci- entific research papers. This tool can save much time for researchers and also readers. Despite the fact that the abstract is just a summary of a research paper, it still costs plenty of time for re- searchers to read through their work and rewrite it into a concise paragraph. Though the text summarization in English has obtained substantial achievements, there has not been much work done in Vietnamese. In recent years, the study of Vietnamese text summarizing algorithms has been mainly on fea- tures ranking (Ung et al., 2015; Nguyen et al., 2016) and graph-based systems (Nguyen-Hoang et al., 2012). Meanwhile, the pre-trained Bidi- rectional Encoder Representation from Trans- formers (BERT) model (Devlin et al., 2019) has shown its significant advantages in many NLP tasks in Vietnamese such as text classifications (Nguyen et al., 2021a; Nguyen et al., 2021b) and machine reading comprehension (Luu et al., 2021). There are some empirical studies on deep learning for this task on Vietnamese datasets, which show outstanding results compared to tra- ditional algorithms (Nguyen et al., 2018). How- ever, there is still a lack of academic work for text summarization in Vietnamese using BERT. In this paper, we aim to examine the influence of several BERT models on extractive text sum- marization in Vietnamese. We conduct exper- iments on both monolingual and multilingual BERT models to compare their performance on this task. Our main contributions on this paper are: • We experiment with new implementations and compare the efficiency of multiple mul- tilingual and monolingual models on gen- erating a summary. As a result, the viB- ERT4news model achieves the highest av- erage ROUGE-1 score …

Content truncated to 500 words