SMTCE: A Social Media Text Classification Evaluation Benchmark and BERTology Models for Vietnamese

SMTCE: A Social Media Text Classification Evaluation Benchmark and BERTology Models for Vietnamese Luan Thanh Nguyen1,2, Kiet Van Nguyen1,2, Ngan Luu-Thuy Nguyen1,2 1Faculty of Information Science and Engineering, University of Information Technology, Ho Chi Minh City, Vietnam 2Vietnam National University, Ho Chi Minh City, Vietnam {luannt, kietnv, ngannlt}@uit.edu.vn Abstract Text classification is a typical natural lan- guage processing or computational linguis- tics task with various interesting applica- tions. As the number of users on social media platforms increases, data acceleration promotes emerging studies on Social Media Text Classification (SMTC) or social me- dia text mining on these valuable resources. In contrast to English, Vietnamese, one of the low-resource languages, is still not con- centrated on and exploited thoroughly. In- spired by the success of the GLUE, we in- troduce the Social Media Text Classification Evaluation (SMTCE) benchmark, as a collec- tion of datasets and models across a diverse set of SMTC tasks. With the proposed bench- mark, we implement and analyze the effec- tiveness of a variety of multilingual BERT- based models (mBERT, XLM-R, and Distilm- BERT) and monolingual BERT-based models (PhoBERT, viBERT, vELECTRA, and viB- ERT4news) for tasks in the SMTCE bench- mark. Monolingual models outperform multi- lingual models and achieve state-of-the-art re- sults on all text classification tasks. It pro- vides an objective assessment of multilingual and monolingual BERT-based models on the benchmarks, which will benefit future studies about BERTology in the Vietnamese language. 1 Introduction With the rise of social media, researching user comments may allow us to understand their be- havior and enhance the quality of cyberspace. So- cial text classification is one of the current popu- lar NLP tasks that aim to solve that problem with an extensive study on comments users left on so- cial media platforms. There are several differ- ent current tasks in social media text classification (SMTC) tasks in several domains, and however, it is designed to be independent of other inconsistent data domains. Motivated by the popularity of the GLUE (Wang et al., 2018), we present a novel Social Media Text Classification Evaluation (SMTCE) benchmark with four social media text classifica- tion tasks in Vietnamese, including constructive speech detection (Nguyen et al., 2021), complaint comment dectection (Nguyen et al., 2022a), emo- tion recognition (Ho et al., 2020), and hate speech detection (Luu et al., 2021) tasks. Natural Language Processing (NLP), a subset of artificial intelligence techniques, is rapidly evolv- ing, with numerous notable accomplishments. The primary goal of assisting computers in understand- ing human language is the bridge for reducing the distance between humans and computers. The current trend in NLP is releasing language mod- els that have been pre-trained on a considerable amount of data to perform various tasks. Hence, we do not need to code from scratch for every task and annotate millions of texts each time. This is the premise for transfer learning and applying the architectures transformers to various downstream tasks in NLP. Bidirectional Encoder Representations from Transformers, known as BERT, have appeared and played an …

Content truncated to 500 words