From FreEM to D’AlemBERT: a Large Corpus and a Language Model for Early Modern French

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 328–337 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 328 ViHealthBERT: Pre-trained Language Models for Vietnamese in Health Text Mining Minh Phuc Nguyen †,D, Vu Hoang Tran †,D, Vu Hoang †, Ta Duc Huy † Trung H. Bui, Steven Q. H. Truong † †VinBrain {v.minhng,v.vutran,v.vuhoang,v,huyta,v.brain01}@vinbrain.net bhtrung@gmail.com Abstract Pre-trained language models have become crucial to achieving competitive results across many Natural Language Processing (NLP) problems. For monolingual pre-trained models in low-resource languages, the quantity has been significantly increased. However, most of them relate to the general domain, and there are limited strong baseline language models for domain-specific. We introduce ViHealthBERT, the first domain-specific pre-trained language model for Vietnamese healthcare. The performance of our model shows strong results while outperforming the general domain language models in all health-related datasets. Moreover, we also present Vietnamese datasets for the healthcare domain for two tasks: Acronym Disambiguation (AD) and Frequently Asked Questions (FAQ) Summarization. We release ViHealthBERT to facilitate future research and downstream applications for Vietnamese NLP in domain-specific. Our dataset and code are available in https://github.com/demdecuong/vihealthbert. Keywords: Low-resource language, language model, healthcare, acronym disambiguation, summarization D denotes equal contribution 1. Introduction Recent large-scale language models show remarkable achievements in key NLP tasks such as Question An- swering (Devlin et al., 2019) and Text Summarization (Raffel et al., 2020). More research studies have dis- covered multilingual language models for better per- formance in a wide range of downstream tasks without considering its language, and lots of low-resource lan- guages benefit from this. (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020; Lan et al., 2020). However, in low-resource languages, monolin- gual language models still demonstrate their superior- ity on general domain benchmarks (Wu and Dredze, 2020). Leveraging mBERT(Devlin et al., 2019) and ELECTRA(Clark et al., 2020) method, viBERT and vELECTRA(Bui et al., 2020) have been proposed and showed superior performance in Vietnamese Named Entity Recognition (NER) and POS Tagging tasks. Then PhoBERT (Nguyen and Tuan Nguyen, 2020) in- troduces the first large-scale BERT-based Vietnamese language model and becomes a strong baseline for Vietnamese NLP tasks, which utilizes a high-volume corpus of news following RoBERTa(Liu et al., 2019) training strategy. While domain-specific language models are abundant (Lee et al., 2020; Rasmy et al., 2021; Roy and Pan, 2021; Gu et al., 2021), studies addressing Vietnamese domain-specific tasks are few and far between, as ex- isting Vietnamese language model studies still focus on the general domain. The first and the only is a language model for legal-domain (Chau et al., 2020) that applies for Vietnamese legal text retrieval, but the study meets the problem of limited pre-training dataset and bench- marks. In the healthcare domain, leveraging general do- main pre-trained weights for downstream tasks is not straightforward for a low-resource language like Vietnamese. There are some attempts to popu- late Vietnamese healthcare corpora. The COVID-19 NER (COVID-19 Named Entity Recognition for Viet- namese) dataset (Truong et …

Content truncated to 500 words