Hierarchical Transfer Learning for Multi-label Text Classification

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6295–6300 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6295 Hierarchical Transfer Learning for Multi-label Text Classification Siddhartha Banerjee, Cem Akkaya, Francisco Perez-Sorrosal, Kostas Tsioutsiouliklis Yahoo Research 701 First Avenue Sunnyvale, CA, USA {siddb, cakkaya, fperez, kostas}@verizonmedia.com Abstract Multi-Label Hierarchical Text Classification (MLHTC) is the task of categorizing docu- ments into one or more topics organized in an hierarchical taxonomy. MLHTC can be for- mulated by combining multiple binary classifi- cation problems with an independent classifier for each category. We propose a novel trans- fer learning based strategy, HTrans, where bi- nary classifiers at lower levels in the hier- archy are initialized using parameters of the parent classifier and fine-tuned on the child category classification task. In HTrans, we use a Gated Recurrent Unit (GRU)-based deep learning architecture coupled with attention. Compared to binary classifiers trained from scratch, our HTrans approach results in signifi- cant improvements of 1% on micro-F1 and 3% on macro-F1 on the RCV1 dataset. Our exper- iments also show that binary classifiers trained from scratch are significantly better than single multi-label models. 1 Introduction Two main approaches for Multi-Label Hierarchi- cal Text Classification (MLHTC) have been pro- posed (Tsoumakas and Katakis, 2007): 1. trans- forming the problem to a collection of independent binary classification problems by training a classi- fier for each category 2. training a single multi- label model that can predict all categories for in- stances simultaneously. In a hierarchical taxonomy of categories, de- pendencies exist between parent and child cat- egories that should be exploited when training classifiers. Recent work on MLHTC uses a Deep Graph-based Convolutional Neural Network (DGCNN) (Peng et al., 2018) -based single multi- label model with a recursive regularization com- ponent to model dependencies between parent and child categories. However, multi-label models suf- fer on categories with very few training exam- ples (Krawczyk, 2016) due to data imbalance. Due to a large prediction space (all categories) of multi- label models, it is very difficult to optimize class weights to handle data imbalance. By contrast, binary classifiers provide more flexibility as class weights for each classifier can easily be optimized based on validation metrics. With a reasonable number of categories (few hundreds), collection of binary classifiers are a feasible option to solve MLHTC problems. Influenced by recent progress of transfer learn- ing on Natural Language Processing (NLP) tasks (Howard and Ruder, 2018; Mou et al., 2016), we present HTrans, a Hierarchical Transfer Learning approach. We hypothesize that introduc- ing dependencies between parent and child cate- gories is possible using transfer learning. There- fore, we initialize parameters of the child category classifier from the binary parent category clas- sifier and later fine-tune the model. The trans- fer of parameters can provide a better starting point for the child category classifier than training from scratch using randomly initialized parame- ters. Without any loss of generality, we propose a simple classification model using Gated Recurrent …

Content truncated to 500 words