Long-tailed Extreme Multi-label Text Classification by the Retrieval of Generated Pseudo Label Descriptions

Findings of the Association for Computational Linguistics: EACL 2023, pages 1092–1106 May 2-6, 2023 ©2023 Association for Computational Linguistics Long-tailed Extreme Multi-label Text Classification by the Retrieval of Generated Pseudo Label Descriptions Ruohong Zhang and Yau-Shian Wang ruohongz,yaushiaw@andrew.cmu.edu Yiming Yang yiming@cs.cmu.edu Donghan Yu dyu2@cs.cmu.edu Tom Vu tom.m.vu@gmail.com Likun Lei llei@flexport.com Abstract Extreme Multi-label Text Classification (XMTC) has been a tough challenge in machine learning research and applications due to the sheer sizes of the label spaces and the severe data scarcity problem associated with the long tail of rare labels in highly skewed distributions. This paper addresses the challenge of tail label prediction by leveraging the power of dense neural retrieval model in mapping input documents (as queries) to relevant label descriptions. To further enhance the quality of label descriptions, we propose to generate pseudo label descriptions from a trained bag-of-words (BoW) classifier, which demonstrates better classification performance under severe scarce data conditions. The proposed approach achieves the state-of-the-art (SOTA) performance of overall label prediction on XMTC benchmark datasets and especially outperforms the SOTA models in the tail label prediction. We also provide a theoretical analysis for relating the BoW and neural models w.r.t. performance lower bound. 1 Introduction Extreme multi-label text classification (XMTC) is the task of tagging documents with relevant labels in a very large and often skewed candidate space. It has a wide range of applications, such as assign- ing subject topics to news or Wikipedia articles, tagging keywords for online shopping items, clas- sifying industrial products for tax purposes, etc. The most difficult part in solving the XMTC problem is to train classification models effectively for the rare labels in the long tail of highly skewed distributions, which suffers severely from the lack of sufficient training instances. Efforts addressing this challenge by the text classification community include Bayesian modeling of graphical dependen- cies among labels (Gopal and Yang, 2010; Gopal et al., 2012), novel loss or regularization of label embeddings (Babbar and Schölkopf, 2019a; Wei 0 5000 10000 15000 20000 25000 30000 0% 5% 10% 15% 20% 25% 30% 35% DEPL X-Transformer Figure 1: The classification performance of X- Transformer and DEPL (ours) measured in macro- averaged F1@19 on the Wiki10-31K dataset. et al., 2021), clustering-based algorithms (Chang et al., 2020; Khandagale et al., 2019; Prabhu et al., 2018), and so on. Despite the remarkable pro- gresses made so far, the problem is still very far from being well solved. Figure 1 shows the per- formance of X-Transformer (Chang et al., 2020), one of the state-of-the-art (SOTA) XMTC models, on the Wiki10-31K benchmark dataset (with over 31k labels). The horizontal axis is the ranks of the labels sorted from rare to common and the vertical axis is the text classification performance measured in macro-averaged F1@19 (higher the better) for binned labels (100 labels per bin). The blue curve is the result of X-Transformer, which has the scores close to 0 (worst possible score) for nearly half of the total labels. In other words, SOTA methods in …

Content truncated to 500 words