Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1051–1062 June 6–11, 2021. ©2021 Association for Computational Linguistics 1051 Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning Hui Liu1 Danqing Zhang2 Bing Yin2 Xiaodan Zhu1 1Ingenuity Labs Research Institute & ECE, Queen’s University, Canada {hui.liu, xiaodan.zhu}@queensu.ca 2Amazon.com Inc, Palo Alto, CA, USA {danqinz, alexbyin}@amazon.com Abstract Exploiting label hierarchies has become a promising approach to tackling the zero-shot multi-label text classification (ZS-MTC) prob- lem. Conventional methods aim to learn a matching model between text and labels, us- ing a graph encoder to incorporate label hierar- chies to obtain effective label representations (Rios and Kavuluru, 2018). More recently, pre- trained models like BERT (Devlin et al., 2018) have been used to convert classification tasks into a textual entailment task (Yin et al., 2019). This approach is naturally suitable for the ZS- MTC task. However, pretrained models are un- derexplored in the existing work because they do not generate individual vector representa- tions for text or labels, making it unintuitive to combine them with conventional graph en- coding methods. In this paper, we explore to improve pretrained models with label hierar- chies on the ZS-MTC task. We propose a Rein- forced Label Hierarchy Reasoning (RLHR) ap- proach to encourage interdependence among labels in the hierarchies during training. Mean- while, to overcome the weakness of flat predic- tions, we design a rollback algorithm that can remove logical errors from predictions during inference. Experimental results on three real- life datasets show that our approach achieves better performance and outperforms previous non-pretrained methods on the ZS-MTC task. 1 Introduction Multi-label text classification (MTC) is a basic NLP problem that underlies many real-life appli- cations like product categorization (Partalas et al., 2015) and medical records coding (Du et al., 2019). The labels in the output space are often interde- pendent and in many applications organized in a hierarchy, as shown in the example in Figure 1. A significant challenge for real-life development of MTC applications is severe deficiencies of anno- tated data for each label in the hierarchy, which demands better solutions for zero-shot learning. Root Active Life Shopping Local Services Bike Rentals Sporting Goods Bikes Bike Repair Input: It’s no doubt the best store to get a bike if you want to do bicycling on weekends!!! Labels: Active Life Bike Rentals Shopping Sporting Goods Bikes Wrong Missing Correct Figure 1: An example of label hierarchy and predic- tions with logical errors. Circled labels are model pre- dictions without incorporating label hierarchy. The existing zero-shot learning for multi-label text classification (ZS-MTC) mostly learns a matching model between the feature space of text and the label space (Ye et al., 2020). In order to learn effective representations for labels, a majority of existing work incorporates label hierarchies via a label encoder designed as Graph Neural Networks (GNNs) that can aggregate the neighboring infor- mation for labels (Chalkidis et al., 2020; Lu et …

Content truncated to 500 words