LaoPLM: Pre-trained Language Models for Lao

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6506–6512 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6506 Nankai Lin and Yingwen Fu are the co-first authors. They have worked together and contributed equally to the paper. LaoPLM: Pre-trained Language Models for Lao Nankai Lin1, Yingwen Fu1, Ziyu Yang1, Chuwei Chen1 and Shengyi Jiang1,2 1 Guangdong University of Foreign Studies, Guangzhou, China 2 Guangzhou Key Laboratory of Multilingual Intelligent Processing, Guangzhou, China {20191010004, 20201010002, 20201002958, 20202005086}@gdufs.edu.cn jiangshengyi@163.com Abstract Trained on large corpora, pre-trained language models (PLMs) can capture different levels of concepts in context and hence generate universal language representations. They are benefitcial for multiple downstream natural language processing (NLP) tasks. Although PTMs have been widely used in most NLP applications, especially for high-resource languages such as English, it is under-represented in Lao NLP research. Previous works on Lao have been hampered by the lack of annotated datasets and the sparsity of language resources. In this work, we construct a text classification dataset to alleviate the resource-scarce situation of the Lao language. In addition, we present the first transformer-based PTMs for Lao with four versions: BERT-Small1, BERT-Base2, ELECTRA-Small3, and ELECTRA- Base4. Furthermore, we evaluate them on two downstream tasks: part-of-speech (POS) tagging and text classification. Experiments demonstrate the effectiveness of our Lao models. We release our models and datasets to the community, hoping to facilitate the future development of Lao NLP applications. Keywords: Pre-trained Language Model, Lao, Text Classification, Part-of-speech Tagging 1. Introduction The use of pre-trained language models (PLMs) represented by BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019) in natural language processing (NLP) has achieved great success in multiple areas. PLMs do not rely on any manually annotated training data but help to produce significant performance gains for various NLP tasks, making them recently become extremely popular. BERT- based PLMs could be categorized into two classes: (1) Monolingual models are language-specific models trained in monolingual datasets (Cui et al., 2019; de Vries et. al., 2019; Vu Xuan et al., 2019; Martin et al., 2020; Nguyen and Tuan Nguyen, 2020). However, the success of monolingual models has been primarily limited to high- resource languages represented by English. (2) Multilingual models (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020; Xue et al., 2021) are trained in datasets from multiple languages and simultaneously support downstream tasks for multiple languages. When it comes to Lao language modeling, there are some concerns to the best of our knowledge: • There are currently no monolingual PLMs for Lao, which has brought certain restrictions to the development of Lao language technology. • Many monolingual and multilingual models are only pre-trained on Wikipedia corpus. It is worth noting that Wikipedia data is not representative of general language use. At the same time, the Lao Wiki data size is relatively small, which may negatively impact the performance of the pre-trained models. PLMs can be significantly improved by using more pre-training data …

Content truncated to 500 words