ViDeBERTa: A powerful pre-trained language model for Vietnamese

Findings of the Association for Computational Linguistics: EACL 2023, pages 1071–1078 May 2-6, 2023 ©2023 Association for Computational Linguistics ViDeBERTa: A powerful pre-trained language model for Vietnamese Cong Dao Tran ∗ FPT Software AI Center daotc2@fsoft.com.vn Nhut Huy Pham ∗ FPT Software AI Center huypn10@fsoft.com.vn Anh Nguyen Microsoft anhnguyen@microsoft.com Truong Son Hy † University of California San Diego tshy@ucsd.edu Tu Vu University of Massachusetts Amherst tuvu@cs.umass.edu Abstract This paper presents ViDeBERTa, a new pre-trained monolingual language model for Vietnamese, with three versions - ViDeBERTaxsmall, ViDeBERTabase, and ViDeBERTalarge, which are pre-trained on a large-scale corpus of high-quality and diverse Vietnamese texts using DeBERTa architec- ture. Although many successful pre-trained language models based on Transformer have been widely proposed for the English language, there are still few pre-trained models for Viet- namese, a low-resource language, that perform good results on downstream tasks, especially Question answering. We fine-tune and evaluate our model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering. The empirical results demonstrate that ViDeBERTa with far fewer parameters surpasses the previous state-of-the-art models on multiple Vietnamese-specific natural language understanding tasks. Notably, ViDeBERTabase with 86M parameters, which is only about 23% of PhoBERTlarge with 370M parameters, still performs the same or better results than the previous state-of-the-art model. Our ViDeBERTa models are available at: https://github.com/HySonLab/ViDeBERTa. 1 Introduction In recent years, pre-trained language models (PLMs) and Transformer-based architecture mod- els have been essential in the advancement of Natural Language Processing (NLP). Large-scale Transformer-based pre-trained models with the ca- pacity to derive a contextual representation of the languages in the training data include GPT (Rad- ford et al., 2019; Brown et al., 2020), BERT (De- vlin et al., 2019), RoBERTa (Liu et al., 2019), XL- Net (Yang et al., 2019b), ELECTRA (Clark et al., 2020), T5 (Raffel et al., 2020), and DeBERTa (He ∗: Co-first authors. †: Correspondent author. et al., 2020, 2021). Following pre-training, these models performed at the cutting edge on various downstream NLP tasks (Devlin et al., 2019). The development of pre-trained models in other lan- guages, including Vietnamese (PhoBERT (Nguyen and Nguyen, 2020); ViBERT (Tran et al., 2020); ViT5 (Phan et al., 2022)), and Arabic (Antoun et al., 2021), has been spurred on by the success of pre- trained models in English. In order to enhance per- formance across several languages by learning both general and language-specific representations, mul- tilingual pre-trained models ( XLM-R (Conneau et al., 2020), mT5 (Xue et al., 2021), and mBART (Liu et al., 2020) are also being developed. Most recently, PhoBERT (Nguyen and Nguyen, 2020), the first large pre-trained model for Viet- namese that inherits the RoBERTa (Liu et al., 2019) architecture, has demonstrated the effective- ness of the trained language model compared with current methods modernized in four Vietnamese- specific tasks, including Part of Speech Tagging (POS), Dependency Parsing, Named Entity Recog- nition (NER), and Natural Language Inference (NLI). Nevertheless, there are still rooms to build an improved pre-trained language model for Viet- namese. Firstly, PhoBERT was pre-trained …

Content truncated to 500 words