Investigating Active Learning Sampling Strategies for Extreme Multi Label Text Classification

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4597–4605 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4597 Investigating Active Learning Sampling Strategies for Extreme Multi Label Text Classification Lukas Fromme∗†, Katsiaryna Mirylenka†, Jonas Kuhn∗, Jasmina Bogojeska† ∗Institute of Natural Language Processing (IMS) - University of Stuttgart, †IBM Research - Zurich lukas.fromme@ims.uni-stuttgart.de, kmi@zurich.ibm.com Abstract Large scale, multi-label text datasets with a high number of classes are expensive to annotate, even more so if they belong to specific language domains. In this work, we aim to build classifiers for these datasets using Active Learning in order to reduce the labeling effort. We outline the challenges when dealing with extreme multi-label settings and show the limitations of existing pool-based Active Learning strategies by considering their effectiveness as well as efficiency in terms of computational cost. In addition, we present five multi-label datasets which were compiled from hierarchical classification tasks to serve as benchmarks in the context of extreme multi-label classification for future experiments. Finally, we provide insights into multi-class, multi-label evaluation and present an improved classifier architecture on top of pre-trained transformer language models. Keywords: Active Learning, Text Classification, Multi-Label 1. Introduction In many fields, one could regard text classification as a solved problem. Even before pre-trained language models were available, common classification tasks such as Spam/Ham sorting or Positive/Negative detec- tion could be completed by a variety of machine learn- ing architectures with a very high accuracy. Conse- quently there has been a trend to move toward more complex classification tasks. One such task is com- monly known as Extreme Multi Label Text Classifi- cation (XMTC) - datasets for this task contain large numbers of distinct classes ranging from hundreds to more than a million. We combine XMTC with Active Learning (AL), which is a technique used to reduce an- notation effort by strategically selecting samples from an unlabeled set. The combination of XMTC and AL has not been exposed to much systematic research al- though it is relevant in many practical scenarios. Large, domain-specific datasets are often assigned numerous, fine-grained categories and each text can belong to multiple of them. For example, one can easily imagine a dataset of law texts which is annotated with several hundred different target classes. Such datasets are also very expensive to annotate due to requiring domain ex- perts who will take a lot of time working through all the classes. Therefore, AL is a particularly attractive prospect, albeit not without its own set of challenges. In order to be truly effective, AL uses various strategies in order to select the most profitable samples from the un- labeled set. However, many of these strategies are not effective on all kinds of data or all classification mod- els. In this work we motivate and introduce the task of Active Learning for Extreme Multi-Label Classifica- tion. Our goal is to gain insights into the performance of established AL approaches on datasets with more than 100 distinct classes. …

Content truncated to 500 words