A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2545–2568 June 6–11, 2021. ©2021 Association for Computational Linguistics 2545 A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios Michael A. Hedderich*1, Lukas Lange*1,2, Heike Adel2, Jannik Strötgen2 & Dietrich Klakow1 1Saarland University, Saarland Informatics Campus, Germany 2Bosch Center for Artificial Intelligence, Germany {mhedderich,dietrich.klakow}@lsv.uni-saarland.de {lukas.lange,heike.adel,jannik.stroetgen}@de.bosch.com Abstract Deep neural networks and huge language mod- els are becoming omnipresent in natural lan- guage applications. As they are known for re- quiring large amounts of training data, there is a growing body of work to improve the performance in low-resource settings. Moti- vated by the recent fundamental changes to- wards neural models and the popular pre-train and fine-tune paradigm, we survey promising approaches for low-resource natural language processing. After a discussion about the dif- ferent dimensions of data availability, we give a structured overview of methods that enable learning when training data is sparse. This includes mechanisms to create additional la- beled data like data augmentation and distant supervision as well as transfer learning set- tings that reduce the need for target supervi- sion. A goal of our survey is to explain how these methods differ in their requirements as understanding them is essential for choosing a technique suited for a specific low-resource setting. Further key aspects of this work are to highlight open issues and to outline promising directions for future research. 1 Introduction Most of today’s research in natural language pro- cessing (NLP) is concerned with the processing of 10 to 20 high-resource languages with a special focus on English, and thus, ignores thousands of languages with billions of speakers (Bender, 2019). The rise of data-hungry deep learning systems in- creased the performance of NLP for high resource- languages, but the shortage of large-scale data in less-resourced languages makes their processing a challenging problem. Therefore, Ruder (2019) named NLP for low-resource scenarios one of the four biggest open problems in NLP nowadays. The umbrella term low-resource covers a spec- trum of scenarios with varying resource conditions. * equal contribution It includes work on threatened languages, such as Yongning Na, a Sino-Tibetan language with 40k speakers and only 3k written, unlabeled sen- tences (Adams et al., 2017). Other languages are widely spoken but seldom addressed by NLP re- search. More than 310 languages exist with at least one million L1-speakers each (Eberhard et al., 2019). Similarly, Wikipedia exists for 300 lan- guages.1 Supporting technological developments for low-resource languages can help to increase par- ticipation of the speakers’ communities in a digital world. Note, however, that tackling low-resource settings is even crucial when dealing with popu- lar NLP languages as low-resource settings do not only concern languages but also non-standard do- mains and tasks, for which – even in English – only little training data is available. Thus, the term “lan- guage” in this paper also includes domain-specific language. This importance of low-resource scenarios and the significant changes in NLP …

Content truncated to 500 words