Mind the User! Measures to More Accurately Evaluate the Practical Value of Active Learning Strategies

Proceedings of Recent Advances in Natural Language Processing, pages 996–1006 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_107 996 Mind the User! Measures to More Accurately Evaluate the Practical Value of Active Learning Strategies Julia Romberg Department of Social Sciences Heinrich Heine University Düsseldorf, Germany julia.romberg@hhu.de Abstract One solution to limited annotation budgets is active learning (AL), a collaborative process of human and machine to strategically select a small but informative set of examples. While current measures optimize AL from a pure ma- chine learning perspective, we argue that for a successful transfer into practice, additional criteria must target the second pillar of AL, the human annotator. In text classification, e.g., where practitioners regularly encounter datasets with an increased number of imbal- anced classes, measures like F1 fall short when finding all classes or identifying rare cases is required. We therefore introduce four measures that reflect class-related demands that users place on data acquisition. In a comprehen- sive comparison of uncertainty-based, diversity- based, and hybrid query strategies on six dif- ferent datasets, we find that strong F1 perfor- mance is not necessarily associated with full class coverage. Uncertainty sampling outper- forms diversity sampling in selecting minority classes and covering classes more efficiently, while diversity sampling excels in selecting less monotonous batches. Our empirical findings emphasize that a holistic view is essential when evaluating AL approaches to ensure their use- fulness in practice – the actual, but often over- looked, goal of development. To this end, stan- dard measures for assessing the performance of text classification need to be complemented by such that more appropriately reflect user needs. 1 Introduction A well-known problem in supervised machine learning (ML) is scenarios where there are limited resources (e.g., budget or time) to annotate data. One approach to solving this problem is active learning (AL; Cohn et al. 1996), a collaborative process between human and machine. Through tar- geted query strategies, AL aims to find a minimal subset of examples whose labels provide the most information for fitting a model. In text classification, many applications have been found to benefit from AL, such as sentiment analysis, intent or topic detection (e.g., Li et al., 2012; Zhang and Zhang, 2019; Tong and Koller, 2001). In addition to these task-specific studies, increased efforts have been made to systematically evaluate the performance of AL strategies across different use cases (e.g., Settles, 2011; Siddhant and Lipton, 2018; Ein-Dor et al., 2020). Yet many academic studies ignore crucial real- world factors, leading to flawed assessments of practical utility. Literature has pointed out sev- eral limitations, including: the difficulty of mak- ing a-priori forecasts about the practical value of strategies (Lowell et al., 2019); the fact that ac- tively acquired datasets are often only effective coupled with the respective model (Lowell et al., 2019; Tomanek and Morik, 2011); the need for out-of-distribution generalization (Longpre et al., 2022); taking into account class imbalance that is regularly encountered in real-world text classifica- tion (Ein-Dor et al., 2020); and the consideration of extreme multi-label scenarios (Wertz …

Content truncated to 500 words