PhoBERT: Pre-trained language models for Vietnamese

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1037–1042 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1037 PhoBERT: Pre-trained language models for Vietnamese Dat Quoc Nguyen1 and Anh Tuan Nguyen2,∗ 1VinAI Research, Vietnam; 2NVIDIA, USA v.datnq9@vinai.io, tuananhn@nvidia.com Abstract We present PhoBERT with two versions— PhoBERTbase and PhoBERTlarge—the first public large-scale monolingual language mod- els pre-trained for Vietnamese. Experimental results show that PhoBERT consistently out- performs the recent best pre-trained multilin- gual model XLM-R (Conneau et al., 2020) and improves the state-of-the-art in multi- ple Vietnamese-specific NLP tasks includ- ing Part-of-speech tagging, Dependency pars- ing, Named-entity recognition and Natural lan- guage inference. We release PhoBERT to fa- cilitate future research and downstream appli- cations for Vietnamese NLP. Our PhoBERT models are available at: https://github. com/VinAIResearch/PhoBERT. 1 Introduction Pre-trained language models, especially BERT (Devlin et al., 2019)—the Bidirectional Encoder Representations from Transformers (Vaswani et al., 2017), have recently become extremely pop- ular and helped to produce significant improve- ment gains for various NLP tasks. The success of pre-trained BERT and its variants has largely been limited to the English language. For other languages, one could retrain a language-specific model using the BERT architecture (Cui et al., 2019; de Vries et al., 2019; Vu et al., 2019; Martin et al., 2020) or employ existing pre-trained mul- tilingual BERT-based models (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020). In terms of Vietnamese language modeling, to the best of our knowledge, there are two main con- cerns as follows: • The Vietnamese Wikipedia corpus is the only data used to train monolingual language models (Vu et al., 2019), and it also is the only Viet- namese dataset which is included in the pre- training data used by all multilingual language ∗Work done during internship at VinAI Research. models except XLM-R. It is worth noting that Wikipedia data is not representative of a general language use, and the Vietnamese Wikipedia data is relatively small (1GB in size uncom- pressed), while pre-trained language models can be significantly improved by using more pre- training data (Liu et al., 2019). • All publicly released monolingual and multi- lingual BERT-based language models are not aware of the difference between Vietnamese syl- lables and word tokens. This ambiguity comes from the fact that the white space is also used to separate syllables that constitute words when written in Vietnamese.1 For example, a 6-syllable written text “Tôi là một nghiên cứu viên” (I am a researcher) forms 4 words “TôiI làam mộta nghiên_cứu_viênresearcher”. Without doing a pre-process step of Vietnamese word segmentation, those models directly apply Byte-Pair encoding (BPE) methods (Sennrich et al., 2016; Kudo and Richardson, 2018) to the syllable-level Vietnamese pre-training data.2 In- tuitively, for word-level Vietnamese NLP tasks, those models pre-trained on syllable-level data might not perform as good as language models pre-trained on word-level data. To handle the two concerns above, we train the first large-scale monolingual BERT-based “base” and “large” models using a 20GB word-level Viet- namese corpus. …

Content truncated to 500 words