Hierarchical Verbalizer for Few-Shot Hierarchical Text Classification

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2918–2933 July 9-14, 2023 ©2023 Association for Computational Linguistics Hierarchical Verbalizer for Few-Shot Hierarchical Text Classification Ke Ji1,2∗, Yixin Lian2, Jingsheng Gao2, Baoyuan Wang2† 1 School of Computer Science and Engineering, Southeast University, China 2 Xiaobing.AI keji@seu.edu.cn {lianyixin, gaojingsheng, wangbaoyuan}@xiaobing.ai Abstract Due to the complex label hierarchy and inten- sive labeling cost in practice, the hierarchical text classification (HTC) suffers a poor perfor- mance especially when low-resource or few- shot settings are considered. Recently, there is a growing trend of applying prompts on pre- trained language models (PLMs), which has exhibited effectiveness in the few-shot flat text classification tasks. However, limited work has studied the paradigm of prompt-based learn- ing in the HTC problem when the training data is extremely scarce. In this work, we define a path-based few-shot setting and establish a strict path-based evaluation metric to further ex- plore few-shot HTC tasks. To address the issue, we propose the hierarchical verbalizer ("Hi- erVerb"), a multi-verbalizer framework treating HTC as a single- or multi-label classification problem at multiple layers and learning vec- tors as verbalizers constrained by hierarchical structure and hierarchical contrastive learning. In this manner, HierVerb fuses label hierarchy knowledge into verbalizers and remarkably out- performs those who inject hierarchy through graph encoders, maximizing the benefits of PLMs. Extensive experiments on three pop- ular HTC datasets under the few-shot settings demonstrate that prompt with HierVerb signif- icantly boosts the HTC performance, mean- while indicating an elegant way to bridge the gap between the large pre-trained model and downstream hierarchical classification tasks. 1 1 Introduction Hierarchical text classification (HTC) is a long- standing research problem due to the wide range of real applications (Mao et al., 2019). However, prior works could still suffer poor performance in practice due to the nature of its sophisticated ∗Work done during an internship at Xiaobing.AI †Corresponding Author 1Our code and few-shot dataset are publicly available at https://github.com/1KE-JI/HierVerb. Fine-tuning paradigm Prompt tuning paradigm Pre-trained Model Input Text + Prompt Multi-label MLM Pre-trained Model Input Text + Prompt Hierarchy-aware Verbalizer Multi-label MLM Label Hierarchy Label Hierarchy (a) Previous Methods Input Text Pre-trained Model Multiple Binary Classification Label Hierarchy (b) HPT (c) Our HierVerb Figure 1: Illustration of methods for HTC problems. (a) Previous methods typically regard HTC as a down- stream classification fine-tuning task. (b) HPT (Wang et al., 2022b) formulates HTC as a multi-label MLM problem following the prompt tuning paradigm. (c) Our HierVerb leverages hierarchy-aware verbalizers, which are more effective for few-shot tuning. label hierarchy as well as the requirement of large- scale data annotation before training the model. Therefore, solving the HTC under the low-resource (Wang et al., 2022b) or few-shot setting becomes an urgent research topic. Existing state-of-the-art HTC models focus on inserting label hierarchy features through graph encoders and then fuse the features into the input layer (Wang et al., 2022b) or output layer (Zhou et al., 2020) of a text encoder such …

Content truncated to 500 words