Meta Learning for Natural Language Processing: A Survey

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 666 - 684 July 10-15, 2022 ©2022 Association for Computational Linguistics Meta Learning for Natural Language Processing: A Survey Hung-yi Lee National Taiwan University hungyilee@ntu.edu.tw Shang-Wen Li Amazon AI∗ shangwel@amazon.com Ngoc Thang Vu University of Stuttgart thangvu@ims.uni-stuttgart.de Abstract Deep learning has been the mainstream tech- nique in natural language processing (NLP) area. However, the techniques require many labeled data and are less generalizable across domains. Meta-learning is an arising field in machine learning studying approaches to learn better learning algorithms. Approaches aim at improving algorithms in various as- pects, including data efficiency and generaliz- ability. Efficacy of approaches has been shown in many NLP tasks, but there is no systematic survey of these approaches in NLP, which hin- ders more researchers from joining the field. Our goal with this survey paper is to offer researchers pointers to relevant meta-learning works in NLP and attract more attention from the NLP community to drive future innovation. This paper first introduces the general concepts of meta-learning and the common approaches. Then we summarize task construction settings and application of meta-learning for various NLP problems and review the development of meta-learning in NLP community. 1 Introduction Recently, deep learning (DL) based natural lan- guage processing (NLP) has been one of the re- search mainstreams and yields significant perfor- mance improvement in many NLP problems. How- ever, DL models are data-hungry. The downside limits such models’ application to different do- mains, languages, countries, or styles because col- lecting in-genre data for model training are costly. To address the challenges, meta-learning tech- niques are gaining attention. Meta-learning, or Learning to Learn, aims to learn better learning algorithms, including better parameter initialization (Finn et al., 2017), optimization strategy (Andrychowicz et al., 2016; Ravi and Larochelle, 2017), network architecture (Zoph ∗Work done while working at Amazon Inc. The current affiliation is Meta AI. and Le, 2017; Zoph et al., 2018; Pham et al., 2018a), distance metrics (Vinyals et al., 2016; Gao et al., 2019a; Sung et al., 2018), and be- yond (Mishra et al., 2018). Meta-learning allows faster fine-tuning, converges to better performance, yields more generalizable models, and it achieves outstanding results for few-shot image classifi- caition (Triantafillou et al., 2020). The benefits alleviate the dependency of learning algorithms on labels and make model development more scalable. Image processing is one of the machine learning areas with abundant applications and established most of the examples in the previous survey papers on meta-learning (Hospedales et al., 2021; Huis- man et al., 2021). On the other hand, there are works showing ben- efits of meta-learning techniques in performance and data efficiency via applying meta-learning to NLP problems. Please refer to Tables 2 and 3 in the appendix for NLP applications improved by meta- learning. Tutorial (Lee et al., 2021b) and Work- shop (Lee et al., 2021a) are organized at ACL 2021 to encourage exchange and collaboration among NLP researchers interested …

Content truncated to 500 words