Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 769–782 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 769 Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling Muhammad Khalifa 1, Muhammad Abdul-Mageed2, Khaled Shaalan3 1Department of Computer Science, Cairo University 2Natural Language Processing Lab, University of British Columbia 3The British University in Dubai m.khalifa@grad.fci-cu.edu.eg muhammad.mageed@ubc.ca, khaled.shaalan@buid.ac.ae Abstract A sufficient amount of annotated data is usu- ally required to fine-tune pre-trained language models for downstream tasks. Unfortunately, attaining labeled data can be costly, especially for multiple language varieties and dialects. We propose to self-train pre-trained language models in zero- and few-shot scenarios to im- prove performance on data-scarce varieties us- ing only resources from data-rich ones. We demonstrate the utility of our approach in the context of Arabic sequence labeling by using a language model fine-tuned on Modern Stan- dard Arabic (MSA) only to predict named en- tities (NE) and part-of-speech (POS) tags on several dialectal Arabic (DA) varieties. We show that self-training is indeed powerful, im- proving zero-shot MSA-to-DA transfer by as large as �10% F1 (NER) and 2% accuracy (POS tagging). We acquire even better per- formance in few-shot scenarios with limited amounts of labeled data. We conduct an ab- lation study and show that the performance boost observed directly results from training data augmentation possible with DA examples via self-training. This opens up opportuni- ties for developing DA models exploiting only MSA resources. Our approach can also be ex- tended to other languages and tasks.1 1 Introduction Neural language models (Xu and Rudnicky, 2000; Bengio et al., 2003) with vectorized word represen- tations (Mikolov et al., 2013) are currently core to a very wide variety of NLP tasks. In specific, using representations from transformer-based (Vaswani et al., 2017) language models (Devlin et al., 2018; Liu et al., 2019), pre-trained on large amounts of unlabeled data and then fine-tuned on labeled task- specific data, has become a popular approach for at:1Our code and fine-tuned models can be accessed https://github.com/mohammadKhalifa/ zero-shot-arabic-dialects improving downstream task performance. This pre-training then fine-tuning scheme has been suc- cessfully applied to several tasks, including ques- tion answering (Yang et al., 2019), social meaning detection (Abdul-Mageed et al., 2020d), text classi- fication (Liu et al., 2019), named entity recognition (NER), and part-of-speech (POS) tagging (Tsai et al., 2019; Conneau et al., 2019). The same setup also works well for cross-lingual learning (Lample and Conneau, 2019; Conneau et al., 2019). Given that it is very expensive to glean labeled resources for all language varieties and dialects, a question arises: “How can we leverage resource- rich dialects to develop models nuanced to down- stream tasks for resource-scarce ones?”. In this work, we aim to answer this particular question by applying self-training to unlabeled target dialect data. We empirically show that self-training is in- deed an effective strategy in zero-shot (where no gold dialectal data are included in training set, Sec- tion 4.2) and few-shot (where …

Content truncated to 500 words