Low-Resource Text Classification via Cross-lingual Language Model Fine-tuning

Low-Resource Text Classification via Cross-lingual Language Model Fine-tuning Xiuhong Li Xinjiang University xjulxh@xju.edu.cn Zhe Li Xinjiang University lizhe@stu.xju.edu.cn Jiabao Sheng Xinjiang University jiabao@stu.xju.edu.cn Wushour Slamu Xinjiang University wushour@xju.edu.cn Abstract Text classification tends to be difficult when data are inadequate considering the amount of man- ually labeled text corpora. For low-resource agglutinative languages including Uyghur, Kazakh, and Kyrgyz (UKK languages), in which words are manufactured via stems concatenated with several suffixes and stems are used as the representation of text content, this feature allows in- finite derivatives vocabulary that leads to high uncertainty of writing forms and huge redundant features. There are major challenges of low-resource agglutinative text classification the lack of labeled data in a target domain and morphologic diversity of derivations in language structures. It is an effective solution which fine-tuning a pre-trained language model to provide meaningful and favorable-to-use feature extractors for downstream text classification tasks. To this end, we propose a low-resource agglutinative language model fine-tuning AgglutiFiT, specifically, we build a low-noise fine-tuning dataset by morphological analysis and stem extraction, then fine- tune the cross-lingual pre-training model on this dataset. Moreover, we propose an attention- based fine-tuning strategy that better selects relevant semantic and syntactic information from the pre-trained language model and uses those features on downstream text classification tasks. We evaluate our methods on nine Uyghur, Kazakh, and Kyrgyz classification datasets, where they have significantly better performance compared with several strong baselines. 1 Introduction Text classification is the backbone of most natural language processing tasks such as sentiment anal- ysis, classification of news topics, and intent recognition. Although deep learning models have reached the most advanced level on many Natural Language Processing(NLP) tasks, these models are trained from scratch, which makes them require larger datasets. Still, many low-resource languages lack rich annotated resources that support various tasks in text classification. For UKK languages, words are de- rived from stem affixes, so there is a huge vocabulary. Stems represent of text content and affixes provide semantic and grammatical functions. Diversity of morphological structure leads to transcribe speech as they pronounce while writing and suffer from high uncertainty of writing forms on these languages which causes the personalized spelling of words especially less frequent words and terms Ablimit et al. (2017). Data collected from the Internet are noisy and uncertain in terms of coding and spelling Ablimit et al. (2016). The main problems in NLP tasks for UKK languages are uncertainty in terms of spelling and coding and annotated datasets inadequate poses a big challenge for classifying short and noisy text data. Data augmentation can effectively solve the problem of insufficient marker corpus in low-resource language datasets. S¸ahin and Steedman (2019) present two simple text augmentation techniques using “crops” sentences by removing dependency links, and “rotates” sentences by moving the tree fragments around the root. However, this may not be sufficient for several other tasks such as cross-language text classification due to irregularities across UKK languages in these kinds of scenarios. Pre-trained language models such as BERT Devlin …

Content truncated to 500 words