Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7109 - 7119 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, Houfeng Wang∗ Key Laboratory of Computational Linguistics, Peking University, MOE, China {wangzh9969, wangpeiyi9979}@gmail.com {hlz, sunx5, wanghf}@pku.edu.cn Abstract Hierarchical text classification is a challeng- ing subtask of multi-label classification due to its complex label hierarchy. Existing meth- ods encode text and label hierarchy separately and mix their representations for classification, where the hierarchy remains unchanged for all input text. Instead of modeling them sep- arately, in this work, we propose Hierarchy- guided Contrastive Learning (HGCLR) to di- rectly embed the hierarchy into a text en- coder. During training, HGCLR constructs positive samples for input text under the guid- ance of the label hierarchy. By pulling to- gether the input text and its positive sample, the text encoder can learn to generate the hierarchy-aware text representation indepen- dently. Therefore, after training, the HGCLR enhanced text encoder can dispense with the redundant hierarchy. Extensive experiments on three benchmark datasets verify the effec- tiveness of HGCLR. 1 Introduction Hierarchical Text Classification (HTC) aims to cat- egorize text into a set of labels that are organized in a structured hierarchy (Silla and Freitas, 2011). The taxonomic hierarchy is commonly modeled as a tree or a directed acyclic graph, in which each node is a label to be classified. As a subtask of multi-label classification, the key challenge of HTC is how to model the large-scale, imbalanced, and structured label hierarchy (Mao et al., 2019). The existing methods of HTC have variously in- troduced hierarchical information. Among recent researches, the state-of-the-art models encode text and label hierarchy separately and aggregate two representations before being classified by a mixed feature (Zhou et al., 2020; Deng et al., 2021). As de- noted in the left part of Figure 1, their main goal is to sufficiently interact between text and structure to *Corresponding author. Text Encoder Graph Encoder Taxonomic hierarchy Joseph Bruno, the State Senate majority leader, ... Text Mixed text-label representation (a) Text feature Label feature Text Encoder Graph Encoder Taxonomic hierarchy Joseph Bruno, the State Senate majority leader, ... Text Mixed text-label representation (a) Text feature Label feature Hierarchy-aware text representation Text Encoder Graph Encoder Taxonomic hierarchy Joseph Bruno, the State Senate majority leader, ... Text (b) Label feature Both training and testing Only training Both training and testing Only training Figure 1: Two ways of introducing hierarchy informa- tion. (a) Previous work model text and labels separately and find a mixed representation. (b) Our method incor- porating hierarchy information into text encoder for a hierarchy-aware text representation. achieve a mixed representation (Chen et al., 2021), which is highly useful for classification (Chen et al., 2020a). However, since the label hierarchy remains unchanged for all text inputs, the graph encoder provides exactly the same representation regardless of the …

Content truncated to 500 words