UCAS-IIE-NLP at SemEval-2023 Task 12: Enhancing Generalization of Multilingual BERT for Low-resource Sentiment Analysis

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1849–1857 July 13-14, 2023 ©2023 Association for Computational Linguistics UCAS-IIE-NLP at SemEval-2023 Task 12: Enhancing Generalization of Multilingual BERT for Low-resource Sentiment Analysis Dou Hu1,2 and Lingwei Wei1,2 and Yaxin Liu1,2 and Wei Zhou1 and Songlin Hu1,2 1 Institute of Information Engineering, Chinese Academy of Sciences 2 School of Cyber Security, University of Chinese Academy of Sciences {hudou, weilingwei, liuyaxin, zhouwei, husonglin}@iie.ac.cn Abstract This paper describes our system designed for SemEval-2023 Task 12: Sentiment analysis for African languages. The challenge faced by this task is the scarcity of labeled data and linguistic resources in low-resource settings. To alleviate these, we propose a generalized multilingual system SACL-XLMR for sentiment analysis on low-resource languages. Specifically, we design a lexicon-based multilingual BERT to facilitate language adaptation and sentiment- aware representation learning. Besides, we ap- ply a supervised adversarial contrastive learn- ing technique to learn sentiment-spread struc- tured representations and enhance model gen- eralization. Our system achieved competitive results, largely outperforming baselines on both multilingual and zero-shot sentiment classifica- tion subtasks. Notably, the system obtained the 1st rank on the zero-shot classification subtask in the official ranking. Extensive experiments demonstrate the effectiveness of our system. 1 Introduction Sentiment analysis is a critical aspect of natural language processing with numerous applications, including public opinion monitoring (Boon-Itt and Skunkan, 2020), healthcare services (Zunic et al., 2020), and recommendation systems (Hu et al., 2021b). However, performing sentiment analysis in low-resource languages poses significant chal- lenges, including the scarcity of labeled data and linguistic resources, as well as the diversity of lan- guages and dialects (Lo et al., 2017; Oueslati et al., 2020). In SemEval-2023 Task 12 (Muhammad et al., 2023b), the focus is on sentiment analysis for African languages in Twitter, which further exac- erbates the challenges due to the presence of tone, code-switching, and digraphia phenomena (Ade- bara and Abdul-Mageed, 2022). Although multilingual pre-trained language mod- els (multilingual PTMs) (Conneau and Lample, 2019; Conneau et al., 2020) have shown potential in cross-lingual transfer learning compared to mono- lingual PTMs (Devlin et al., 2019; Hu et al., 2022a), they have limitations in capturing nuances and cul- tural differences within a language, especially in the context of dialects and regional variations. In this paper, we propose a generalized multilin- gual system named SACL-XLMR to address these limitations and enhance the generalization of mul- tilingual PTMs for under-represented languages, particularly African languages. Our system lever- ages a lexicon-based multilingual BERT model to facilitate language adaptation and sentiment- aware representation learning. Additionally, we apply a supervised adversarial contrastive learn- ing (SACL) technique (Hu et al., 2023) to learn sentiment-spread structured representations and en- hance model generalization. We present the details of the proposed sys- tem and evaluate its performance on SemEval- 2023 Task 12. Our system achieves remarkable performance, outperforming baselines by +1.1% weighted-F1 score on multilingual sentiment classi- fication subtask and by +2.8% weighted-F1 score on zero-shot sentiment classification subtask in the AfriSenti-SemEval datasets (Muhammad et al., …

Content truncated to 500 words