Knowledge-Aware Meta-learning for Low-Resource Text Classification

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1814–1821 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1814 Knowledge-Aware Meta-learning for Low-Resource Text Classification Huaxiu Yao1†, Yingxin Wu2‡, Maruan Al-Shedivat4§, Eric P. Xing3,4§ 1Stanford University; 2University of Science and Technology of China 3Mohamed bin Zayed University of Artificial Intelligence, 4Carnegie Mellon University †huaxiu@cs.stanford.edu, ‡wuyxinsh@gmail.com §{alshedivat, epxing}@cs.cmu.edu Abstract Meta-learning has achieved great success in leveraging the historical learned knowledge to facilitate the learning process of the new task. However, merely learning the knowl- edge from the historical tasks, adopted by cur- rent meta-learning algorithms, may not gener- alize well to testing tasks when they are not well-supported by training tasks. This paper studies a low-resource text classification prob- lem and bridges the gap between meta-training and meta-testing tasks by leveraging the exter- nal knowledge bases. Specifically, we propose KGML to introduce additional representation for each sentence learned from the extracted sentence-specific knowledge graph. The exten- sive experiments on three datasets demonstrate the effectiveness of KGML under both super- vised adaptation and unsupervised adaptation settings. 1 Introduction Learning-to-learn (or meta-learning) (Bengio et al., 1990; Schmidhuber, 1992; Hochreiter et al., 2001; Vinyals et al., 2016; Finn et al., 2017) has recently emerged as a successful technique for training mod- els on large collections of low-resource tasks. In the natural language domain, it has been used to improve machine translation (Gu et al., 2018), se- mantic parsing (Sun et al., 2020), text classification (Bao et al., 2019; Geng et al., 2020, 2019; Li et al., 2020), sequence labelling (Li et al., 2021), text generation (Guo et al., 2020), knowledge graph reasoning (Wang et al., 2019), among many other applications in low-resource settings. Meta-learning has been shown to dominate self- supervised pretraining techniques such as masked language modeling (Devlin et al., 2018) when the training tasks are representative enough of the tasks encountered at test time (Bansal et al., 2019, 2020). However, in practice, it requires access to a very large number of training tasks (Al-Shedivat et al., love sweetheart lover women soul supporter help Sentence: Thank you for the love and help, sweetheart derivational hypernym KNN link hypernym derivational hypernym Entity 1 Entity 2 Entity 3 Sentence-specific KG: Red color: Original Entities Blue color: Extracted Entities Figure 1: Illustration of extracting a sentence-specific KG from a shared KB. 2021) and, especially in the natural language do- main, mitigating discrepancy between training and test tasks becomes non-trivial due to new concepts or entities that can be present at test time only. In this paper, we propose to leverage external knowledge bases (KBs) in order to bridge the gap between the training and test tasks and enable more efficient meta-learning for low-resource text clas- sification. Our key idea is based on computing additional representations for each sentence by con- structing and embedding sentence-specific knowl- edge graphs (KGs) of entities extracted from a knowledge base shared across all tasks (e.g., Fig. 1). These representations are computed using a graph neural network (GNN) …

Content truncated to 500 words