HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text Classification

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3740 - 3751 December 7-11, 2022 ©2022 Association for Computational Linguistics HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text Classification Zihan Wang1† Peiyi Wang1† Tianyu Liu2 Binghuai Lin2 Yunbo Cao2 Zhifang Sui1 Houfeng Wang1∗ 1 MOE Key Laboratory of Computational Linguistics, Peking University, China 2 Tencent Cloud Xiaowei {wangzh9969, wangpeiyi9979}@gmail.com; {szf, wanghf}@pku.edu.cn {rogertyliu, binghuailin, yunbocao}@tencent.com; Abstract Hierarchical text classification (HTC) is a chal- lenging subtask of multi-label classification due to its complex label hierarchy. Recently, the pretrained language models (PLM) have been widely adopted in HTC through a fine- tuning paradigm. However, in this paradigm, there exists a huge gap between the classifi- cation tasks with sophisticated label hierarchy and the masked language model (MLM) pre- training tasks of PLMs and thus the potential of PLMs cannot be fully tapped. To bridge the gap, in this paper, we propose HPT, a Hierarchy-aware Prompt Tuning method to handle HTC from a multi-label MLM perspec- tive. Specifically, we construct a dynamic vir- tual template and label words that take the form of soft prompts to fuse the label hierar- chy knowledge and introduce a zero-bounded multi-label cross-entropy loss to harmonize the objectives of HTC and MLM. Extensive ex- periments show HPT achieves state-of-the-art performances on 3 popular HTC datasets and is adept at handling the imbalance and low resource situations. Our code is available at https://github.com/wzh9969/HPT. 1 Introduction Hierarchical text classification (HTC) aims to cate- gorize a text into a set of labels with a structured class hierarchy (commonly modeled as a tree) (Silla and Freitas, 2011). HTC is a multi-label text clas- sification problem, where the classification result corresponds to one or more paths of the hierarchy (Zhou et al., 2020). The major challenge of HTC is to model the large-scale, imbalanced, and struc- tured label hierarchy (Mao et al., 2019). As shown in Figure 1(a), existing state-of-the- art HTC models (Zhou et al., 2020; Deng et al., 2021; Chen et al., 2021; Zhao et al., 2021) sepa- rately extract text and label hierarchy features by †Equal contribution. *Corresponding author. Text PLM Label Hierarchy Mixed Feature (a) Previous Methods (b) Ours Graph Encoder Multiple Binary Classification Text Label Hierarchy Hierarchy-aware Template Words Hierarchy–aware Label Words Multi-label MLM PLM Figure 1: Comparison of previous methods and our HPT. (a) Previous models formulate HTC as a multiple binary classification problem and utilize the PLM in a fine-tuning paradigm. (b) HPT follows a prompt tuning paradigm that transforms HTC into a hierarchy-aware multi-label MLM problem. utilizing text and graph encoders, and then fuse the two sources of features into a final represen- tation for text classification. Specifically, Chen et al. (2021) takes advantage of powerful pretrained language models (PLMs) in HTC through a fine- tuning paradigm, where they use PLMs as the text encoder. In this paradigm, the PLMs are trained to infer with complex label hierarchy. Despite the success of the fine-tuning paradigm, some recent studies suggest that it may suffer …

Content truncated to 500 words