Extreme Zero-Shot Learning for Extreme Text Classification

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5455 - 5468 July 10-15, 2022 ©2022 Association for Computational Linguistics Extreme Zero-Shot Learning for Extreme Text Classification Yuanhao Xiong∗ UCLA yhxiong@cs.ucla.edu Wei-Cheng Chang Amazon Inc. weicheng.cmu@gmail.com Cho-Jui Hsieh UCLA chohsieh@cs.ucla.edu Hsiang-Fu Yu Amazon Inc. rofu.yu@gmail.com Inderjit Dhillon UT Austin inderjit@cs.utexas.edu Abstract The eXtreme Multi-label text Classification (XMC) problem concerns finding most relevant labels for an input text instance from a large label set. However, the XMC setup faces two challenges: (1) it is not generalizable to predict unseen labels in dynamic environments, and (2) it requires a large amount of supervised (instance, label) pairs, which can be difficult to obtain for emerging domains. In this paper, we consider a more practical scenario called Extreme Zero-Shot XMC (EZ-XMC), in which no supervision is needed and merely raw text of instances and labels are accessible. Few-Shot XMC (FS-XMC), an extension to EZ-XMC with limited supervision is also investigated. To learn the semantic embeddings of instances and labels with raw text, we propose to pre-train Transformer-based en- coders with self-supervised contrastive losses. Specifically, we develop a pre-training method MACLR, which thoroughly leverages the raw text with techniques including Multi-scale Adaptive Clustering, Label Regularization, and self-training with pseudo positive pairs. Experimental results on four public EZ-XMC datasets demonstrate that MACLR achieves superior performance compared to all other leading baseline methods, in particular with approximately 5-10% improvement in precision and recall on average. Moreover, we show that our pre-trained encoder can be further improved on FS-XMC when there are a limited number of ground-truth positive pairs in training. Our code is available at https://github.com/amzn/pecos/ tree/mainline/examples/MACLR. 1 Introduction The eXtreme Multi-label text Classification (XMC) problem aims at tagging a text input with most rele- vant subset of labels from an extremely large output space. Many web-related applications can be for- mulated as an XMC task with encouraging results, ∗Work done during an internship at Amazon Inc. such as finding the best matching products from a large catalog in e-commerce systems (Medini et al., 2019; Chang et al., 2021), auto-completing queries given its prefix on search engines (Yadav et al., 2021), predicting search keywords for dynamic ad- vertising (Prabhu et al., 2018; Chang et al., 2020b), tagging categories of Wikipedia articles from a large label taxonomy (Dekel and Shamir, 2010; Chalkidis et al., 2019), to name just a few. The current XMC setup is built on full label coverage and full supervision, where full label cov- erage means labels to be predicted have appeared in the training set and full supervision indicates it re- quires a significant number of annotated (instance, label) pairs. In detail, it is assumed that an XMC algorithm has access to raw text of instances and labels, together with their corresponding relations during training, as shown in Figure 1. However, there are several limitations of this XMC setting. First of all, due to the assumption of full label coverage, it is typical …

Content truncated to 500 words