Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3135 - 3151 December 7-11, 2022 ©2022 Association for Computational Linguistics Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing Abbas Ghaddar1,∗ Yimeng Wu1,∗ Sunyam Bagga1 Ahmad Rashid1 Khalil Bibi1 Mehdi Rezagholizadeh1 Chao Xing1 Yasheng Wang1 Duan Xinyu2 Zhefeng Wang2 Baoxing Huai2 Xin Jiang1 Qun Liu1 and Philippe Langlais3 1 Huawei Technologies Co., Ltd. 2 Huawei Cloud Computing Technologies Co., Ltd 3 RALI/DIRO, Université de Montréal, Canada {abbas.ghaddar,yimeng.wu,sunyam.bagga}@huawei.com {ahmad.rashid,khalil.bibi,mehdi.rezagholizadeh}@huawei.com {xingchao.ml,wangyasheng,duanxinyu}@huawei.com {wangzhefeng,huaibaoxing,jiang.xin,qun.liu}@huawei.com felipe@iro.umontreal.ca Abstract There is a growing body of work in recent years to develop pre-trained language models (PLMs) for the Arabic language. This work addresses two major problems in existing Arabic PLMs that limit the progress of the Arabic NLU and NLG fields. First, existing Arabic PLMs are not well-explored and their pre-training can be improved significantly using a more methodical approach. Second, there is a lack of systematic and reproducible evaluation of these models in the literature. We revisit both the pre-training and evaluation of Arabic PLMs. In terms of pre-training, we explore the impact of the qual- ity of the pretraining data, the size of the model and the incorporation of character-level infor- mation to Arabic PLMs. As a result, we release three new Arabic BERT-style models (JABER, Char-JABER, and SABER), and two T5-style models (AT5S and AT5B). In terms of eval- uation, we conduct a comprehensive empiri- cal study to systematically evaluate the perfor- mance of existing state-of-the-art models on ALUE, a leaderboard-powered benchmark for Arabic NLU tasks, and on a subset of Arabic generative tasks. We show that our models significantly outperform existing Arabic PLMs and achieve a new state-of-the-art performance on both discriminative and generative tasks. 1 Introduction Pre-trained language models (PLMs) such as BERT (Devlin et al., 2018), GPT (Radford et al., 2018), and T5 (Raffel et al., 2019) have become the default standard architectures for modern natu- ral language understanding (NLU) systems in both academic (Kalyan et al., 2021; Min et al., 2021) and industrial settings (Chakravarti et al., 2020; ∗ Equal contribution Tunstall et al., 2022; Li et al., 2021). On the evalu- ation side, the community has widely adopted the leaderboard paradigm1 as a reliable and fair tool to track the progress on various NLP tasks (Mehri et al., 2020; Wang et al., 2018, 2019). Recent years have seen tremendous efforts to develop language-specific PLMs (Le et al., 2020; Chan et al., 2020; Canete et al., 2020; Ulˇcar and Robnik-Šikonja, 2020) and leaderboards (Xu et al., 2020, 2021; Shavrina et al., 2020; Wilie et al., 2020) for languages other than English. These language-specific PLMs have proven to be more accurate than multilingual ones in monolingual evaluation settings (Martin et al., 2019; Wei et al., 2019; Safaya et al., 2020). Moreover, creating high-quality human-curated benchmarks is consid- ered to be of utmost importance for reliable evalua- tion (DeYoung et al., 2020; Kiela et al., 2021). For some high-resource languages like Chinese, the community has been …

Content truncated to 500 words