Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects

Findings of the Association for Computational Linguistics: ACL 2022, pages 1708 - 1719 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects Go Inoue, Salam Khalifa†, and Nizar Habash Computational Approaches to Modeling Language (CAMeL) Lab New York University Abu Dhabi †Stony Brook University {go.inoue,nizar.habash}@nyu.edu salam.khalifa@stonybrook.edu Abstract We present state-of-the-art results on mor- phosyntactic tagging across different varieties of Arabic using fine-tuned pre-trained trans- former language models. Our models consis- tently outperform existing systems in Modern Standard Arabic and all the Arabic dialects we study, achieving 2.6% absolute improvement over the previous state-of-the-art in Modern Standard Arabic, 2.8% in Gulf, 1.6% in Egyp- tian, and 8.3% in Levantine. We explore differ- ent training setups for fine-tuning pre-trained transformer language models, including train- ing data size, the use of external linguistic re- sources, and the use of annotated data from other dialects in a low-resource scenario. Our results show that strategic fine-tuning using datasets from other high-resource dialects is beneficial for a low-resource dialect. Addition- ally, we show that high-quality morphological analyzers as external linguistic resources are beneficial especially in low-resource settings. 1 Introduction Fine-tuning pre-trained language models like BERT (Devlin et al., 2019) has achieved great success in a wide variety of natural language processing (NLP) tasks, e.g., sentiment analy- sis (Abu Farha et al., 2021), question answer- ing (Antoun et al., 2020), named entity recogni- tion (Ghaddar et al., 2022), and dialect identifica- tion (Abdelali et al., 2021). Pre-trained LMs have also been used for enabling technologies such as part-of-speech (POS) tagging (Lan et al., 2020; Khalifa et al., 2021; Inoue et al., 2021) to produce features for downstream processes. Previous POS tagging results using pre-trained LMs focused on core POS tagsets; however, it is still not clear how these models perform on the full morphosyntac- tic tagging task of very morphologically rich lan- guages, where the size of the full tagset can be in the thousands. One such language is Arabic, where lemmas inflect to a large number of forms through different combinations of morphological features and cliticization. Additionally, Arabic orthography omits the vast majority of its optional diacritical marks which increases morphosyntactic ambiguity. A third challenge for Arabic is its numerous vari- ants. Modern Standard Arabic (MSA) is the pri- marily written variety used in formal settings. Di- alectal Arabic (DA), by contrast, is the primarily spoken unstandardized variant. MSA and different DAs, e.g., Gulf (GLF), Egyptian (EGY), and Lev- antine (LEV), vary in terms of their grammar and lexicon to the point of impeding system usability cross-dialectally (Habash et al., 2012). Further- more, these variants currently differ in the degree of data availability: MSA is the highest resourced variant, followed by GLF and EGY, and then LEV. In this paper, we explore different training setups for fine-tuning Arabic pre-trained language models in the complex morphosyntactic tagging task for four Arabic variants (MSA, GLF, EGY, and LEV) under controlled experimental settings. We aim to answer the following …

Content truncated to 500 words