Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting

Findings of the Association for Computational Linguistics: ACL 2022, pages 2441 - 2454 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting Ilias Chalkidis and Anders Søgaard Department of Computer Science, University of Copenhagen, Denmark [ilias.chalkidis,soegaard]@di.ku.dk Abstract In document classification for, e.g., legal and biomedical text, we often deal with hundreds of classes, including very infrequent ones, as well as temporal concept drift caused by the influence of real world events, e.g., policy changes, conflicts, or pandemics. Class im- balance and drift can sometimes be mitigated by resampling the training data to simulate (or compensate for) a known target distribution, but what if the target distribution is determined by unknown future events? Instead of simply resampling uniformly to hedge our bets, we fo- cus on the underlying optimization algorithms used to train such document classifiers and evaluate several group-robust optimization al- gorithms, initially proposed to mitigate group- level disparities. Reframing group-robust algo- rithms as adaptation algorithms under concept drift, we find that Invariant Risk Minimization and Spectral Decoupling outperform sampling- based approaches to class imbalance and con- cept drift, and lead to much better performance on minority classes. The effect is more pro- nounced the larger the label set. 1 Introduction Large-scale multi-label document classification is the task of assigning a subset of labels from a large predefined set – of, say, hundreds or thousands of labels – to a given document. Common applica- tions include labeling scientific publications with concepts from ontologies (Tsatsaronis et al., 2015), associating medical records with diagnostic and procedure labels (Johnson et al., 2017), pairing leg- islation with relevant legal concepts (Mencia and Fürnkranzand, 2007), or categorizing product de- scriptions (Lewis et al., 2004). The task in general presents interesting challenges due to the large la- bel space and two-tiered skewed label distributions. Class Imbalance In multi-label classification, datasets often exhibit class imbalance, i.e., skewed Figure 1: Model performance using random vs. chronological splits across the medium-sized datasets (Table 1). The shaded parts of the bars are the train/test discrepancy due to over-fitting. The performance drop from random to chronological splits demonstrates the temporal concept drift. label distributions (Figure 2). Common methods in- clude resampling and reweighting based on heuris- tic assumptions, but methods are known to suffer from unstable performance, poor applicability, and high computational cost in complex tasks where their assumptions do not hold (Liu et al., 2020). Datasets with long-tail frequency distributions, like the ones considered below – sometimes referred to as power-law datasets (Rubin et al., 2012) – can be particular challenging. Also, the heuristics fix the trade-off between exploiting as much of the training data as possible and balancing the classes, instead of trying to learn the optimal trade-off. Temporal Concept Drift Moreover, class distri- butions may change over time. This is one di- mension of the temporal generalization problem (Lazaridou et al., 2021). Recently, Søgaard et al. (2021) argued chronological data splits are neces- sary to …

Content truncated to 500 words