Characterizing and Measuring Linguistic Dataset Drift

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8953–8967 July 9-14, 2023 ©2023 Association for Computational Linguistics Characterizing and Measuring Linguistic Dataset Drift Tyler A. Chang1∗ Kishaloy Halder2† Neha Anna John2 Yogarshi Vyas2 Yassine Benajiba2 Miguel Ballesteros2 Dan Roth2 1University of California San Diego 2AWS AI Labs tachang@ucsd.edu {kishaloh,nehajohn,yogarshi,benajiy,ballemig,drot}@amazon.com Abstract NLP models often degrade in performance when real world data distributions differ markedly from training data. However, exist- ing dataset drift metrics in NLP have generally not considered specific dimensions of linguistic drift that affect model performance, and they have not been validated in their ability to pre- dict model performance at the individual ex- ample level, where such metrics are often used in practice. In this paper, we propose three di- mensions of linguistic dataset drift: vocabulary, structural, and semantic drift. These dimen- sions correspond to content word frequency divergences, syntactic divergences, and mean- ing changes not captured by word frequencies (e.g. lexical semantic change). We propose interpretable metrics for all three drift dimen- sions, and we modify past performance pre- diction methods to predict model performance at both the example and dataset level for En- glish sentiment classification and natural lan- guage inference. We find that our drift metrics are more effective than previous metrics at pre- dicting out-of-domain model accuracies (mean 16.8% root mean square error decrease), par- ticularly when compared to popular fine-tuned embedding distances (mean 47.7% error de- crease). Fine-tuned embedding distances are much more effective at ranking individual ex- amples by expected performance, but decom- posing into vocabulary, structural, and semantic drift produces the best example rankings of all considered model-agnostic drift metrics (mean 6.7% ROC AUC increase). 1 Introduction Dataset drift, when test data distributions differ from a model’s training data, can have detrimen- tal effects on NLP model performance (Broscheit et al., 2022; Do et al., 2021; Koh et al., 2021). In real world scenarios, models are regularly mon- itored for potential performance degradations by ∗ Work done during an internship at AWS AI Labs. † Corresponding author. comparing incoming test data with the training data (Elango et al., 2022; Nigenda et al., 2022). For these scenarios, researchers have proposed a vari- ety of linguistic dataset drift metrics that aim to predict NLP model performance degradations be- tween training and test domains (Elsahar and Gallé, 2019; Ramesh Kashyap et al., 2021). However, previous drift metrics and performance predictions suffer from several limitations. First, previous metrics have generally been designed as holistic measures of linguistic dataset drift, despite the fact that different NLP tasks and models might be sensitive to different dimensions of linguistic drift. Second, previous research has focused on drift metrics at the dataset level rather than the in- dividual example level. Not only does this require multiple labeled evaluation domain datasets to make out-of-domain performance predictions (re- gressions require multiple dataset-level drift values to fit to; Elsahar and Gallé, 2019; Ramesh Kashyap et al., 2021), but drift metrics are often used …

Content truncated to 500 words