Distributionally Robust Finetuning BERT for Covariate Drift in Spoken Language Understanding

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1970 - 1985 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Distributionally Robust Finetuning BERT for Covariate Drift in Spoken Language Understanding Samuel Broscheit1,2, Quynh Do2, Judith Gaspers2, 1Data and Web Science Research Group, University of Mannheim, Germany 2Amazon Alexa AI Abstract In this study, we investigate robustness against covariate drift in spoken language understand- ing (SLU). Covariate drift can occur in SLU when there is a drift between training and test- ing regarding what users request or how they request it. To study this we propose a method that exploits natural variations in data to create a covariate drift in SLU datasets. Experiments show that a state-of-the-art BERT-based model suffers performance loss under this drift. To mitigate the performance loss, we investigate distributionally robust optimization (DRO) for finetuning BERT-based models. We discuss some recent DRO methods, propose two new variants and empirically show that DRO im- proves robustness under drift. 1 Introduction A common assumption in machine learning is that training data and test data are independent and iden- tically distributed (i.i.d.). Unfortunately, this may not hold in practice and the test distribution might have drifted from the training distribution which can lead to a significant drop of performance in real- world applications (Moreno-Torres et al., 2012). Consider spoken language understanding (SLU), i.e., the task of mapping an utterance to a ma- chine readable semantic interpretation, which is commonly used in voice controlled devices like Alexa, Siri or Google Assistant. Distributional drifts can be caused by seasonal and non-seasonal factors. For example, festive holidays can lead to many requests outside the daily routine. New users might use an uncommon phrasing to express their intent or they might request an uncommon song to be played. Such drifts in the input distribu- tion are referred to as covariate drift. When users’ requests fail to be recognized by the device they might rephrase their intent until they succeed, es- sentially adapting to the SLU model’s distribution. This means that, even when new training samples are drawn from new user utterances, the dominance of the old distribution already present in the SLU model is reinforced. Fine-tuned pre-trained lan- guage models (PLM), such as BERT (Devlin et al., 2019) yield strong performance on SLU bench- marks (Chen et al., 2019). Yet, it has been observed that also PLMs are vulnerable to drifts, and there is a high interest in understanding the robustness of PLMs (Oren et al., 2020a; McCoy et al., 2019; Tu et al., 2020; Cao et al., 2020). The goals of this study are to investigate the impact of covariate drift on BERT’s performance, and to experimentally investigate distributionally robust optimization (DRO) for finetuning BERT. While we focus on sequence classification for SLU, i.e., intent classification (IC) and slot filling (SF), we expect the insights of this study to be applicable also to other sequence classification tasks. To study the impact of covariate drift on …

Content truncated to 500 words