Meta-LMTC: Meta-Learning for Large-Scale Multi-Label Text Classification

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8633–8646 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8633 Meta-LMTC: Meta-Learning for Large-Scale Multi-Label Text Classification Ran Wang, Xi’ao Su, Siyu Long, Xinyu Dai ∗, Shujian Huang, Jiajun Chen National Key Laboratory for Novel Software Technology, Nanjing University, China Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing, China {wangr,nlp_suxa,longsy}@smail.nju.edu.cn {daixinyu,huangsj,chenjj}@nju.edu.cn Abstract Large-scale multi-label text classification (LMTC) tasks often face long-tailed label distri- butions, where many labels have few or even no training instances. Although current methods can exploit prior knowledge to handle these few/zero-shot labels, they neglect the meta- knowledge contained in the dataset that can guide models to learn with few samples. In this paper, for the first time, this problem is ad- dressed from a meta-learning perspective. How- ever, the simple extension of meta-learning approaches to multi-label classification is sub- optimal for LMTC tasks due to long-tailed label distribution and coexisting of few- and zero- shot scenarios. We propose a meta-learning approach named META-LMTC. Specifically, it constructs more faithful and more diverse tasks according to well-designed sampling strategies and directly incorporates the objective of adapt- ing to new low-resource tasks into the meta- learning phase. Extensive experiments show that META-LMTC achieves state-of-the-art per- formance against strong baselines and can still enhance powerful BERTlike models. 1 Introduction Large-scale multi-label text classification (LMTC) is a fundamental and practical task in natural lan- guage processing (Tsoumakas et al., 2010). LMTC can be found in several domains, such as organiz- ing documents in Wikipedia articles (Partalas et al., 2015), annotating medical records with diagnostic and procedure labels (Yan et al., 2010; Rios and Kavuluru, 2018), assigning legislation with rele- vant legal concepts (Chalkidis et al., 2019). Differ- ent from multi-class classification, the LMTC task aims to assign multiple labels from a large prede- fined set (typically thousands) to each instance. Due to the large predefined label set and lim- ited annotated resources, LMTC tasks usually face the challenges of long-tailed label distribution, i.e., ∗ Corresponding author. many labels have few or even no annotated sam- ples. For example, in EURLEX15K (Chalkidis et al., 2019), about 70% of seen labels have been assigned to less than 20 documents (i.e., few-shot labels); and more than 40% of the predefined labels are not associated with any document (i.e., zero- shot labels). To make matter worse, new labels continually emerge as the field evolves. Though few/zero-shot labels may not contribute heavily to the overall performance, correct prediction of such labels is crucial in some cases (Rios and Kavuluru, 2018). For instance, when assigning the diagnosis labels to electronic health records, incorrect pre- dictions of these labels either bring unnecessary financial burdens or make patients ignore potential health risks. These factors require models to utilize few or no samples to accurately assign labels. To cope with these few/zero-shot labels, current models typically match texts to feature vectors for each label obtained by exploiting prior label infor- mation. Specifically, Rios and Kavuluru …

Content truncated to 500 words