ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7088–7105 August 1–6, 2021. ©2021 Association for Computational Linguistics 7088 ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic Muhammad Abdul-Mageed† AbdelRahim Elmadany† El Moatez Billah Nagoudi† Natural Language Processing Lab The University of British Columbia {muhammad.mageed,a.elmadany,moatez.nagoudi}@ubc.ca Abstract Pre-trained language models (LMs) are cur- rently integral to many natural language pro- cessing systems. Although multilingual LMs were also introduced to serve many languages, these have limitations such as being costly at inference time and the size and diversity of non-English data involved in their pre-training. We remedy these issues for a collection of diverse Arabic varieties by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT. To eval- uate our models, we also introduce ARLUE, a new benchmark for multi-dialectal Arabic language understanding evaluation. ARLUE is built using 42 datasets targeting six differ- ent task clusters, allowing us to offer a se- ries of standardized experiments under rich conditions. When fine-tuned on ARLUE, our models collectively achieve new state-of-the- art results across the majority of tasks (37 out of 48 classification tasks, on the 42 datasets). Our best model acquires the highest ARLUE score (77.40) across all six task clusters, out- performing all other models including XLM- RLarge (∼ 3.4× larger size). Our models are publicly available at https://github.com/UBC- NLP/marbert and ARLUE will be released through the same repository. 1 Introduction Language models (LMs) exploiting self-supervised learning such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019a) have recently emerged as powerful transfer learning tools that help im- prove a very wide range of natural language pro- cessing (NLP) tasks. Multilingual LMs such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (XLM-R) (Conneau et al., 2020) have also been introduced, but are usually outperformed by mono- lingual models pre-trained with larger vocabulary and bigger language-specific datasets (Virtanen et al., 2019; Antoun et al., 2020; Dadas et al., 2020; † All authors contributed equally. de Vries et al., 2019; Le et al., 2020; Martin et al., 2020; Nguyen and Tuan Nguyen, 2020). Since LMs are costly to pre-train, it is important to keep in mind the end goals they will serve once developed. For example, (i) in addition to their util- ity on ‘standard’ data, it is useful to endow them with ability to excel on wider real world settings such as in social media. Some existing LMs do not meet this need since they were trained on datasets that do not sufficiently capture the nuances of social media language (e.g., frequent use of abbreviations, emoticons, and hashtags; playful character repeti- tions; neologisms and informal language). It is also desirable to build models able to (ii) serve diverse communities (e.g., speakers of dialects of a given language), rather than focusing only on mainstream varieties. In addition, once created, models should be (iii) usable in energy efficient scenarios. This means that, for example, medium-to-large models with …

Content truncated to 500 words