To Annotate or Not? Predicting Performance Drop under Domain Shift

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2163–2173, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2163 To Annotate or Not? Predicting Performance Drop under Domain Shift Hady Elsahar and Matthias Gall´e NAVER LABS Europe {hady.elsahar,matthias.galle}@naverlabs.com Abstract Performance drop due to domain-shift is an endemic problem for NLP models in produc- tion. This problem creates an urge to con- tinuously annotate evaluation datasets to mea- sure the expected drop in the model perfor- mance which can be prohibitively expensive and slow. In this paper, we study the problem of predicting the performance drop of mod- ern NLP models under domain-shift, in the ab- sence of any target domain labels. We investi- gate three families of methods (H-divergence, reverse classification accuracy and confidence measures), show how they can be used to pre- dict the performance drop and study their ro- bustness to adversarial domain-shifts. Our re- sults on sentiment classification and sequence labelling show that our method is able to pre- dict performance drops with an error rate as low as 2.15% and 0.89% for sentiment analy- sis and POS tagging respectively. 1 Introduction Building Natural Language Processing models that perform well in the wild is still an open and challenging problem. It is well known that mod- ern machine-learning models can be brittle, mean- ing that – even when achieving impressive perfor- mance on the evaluation set – their performance can degrade significantly when exposed to new ex- amples with differences in vocabulary and writ- ing style (Blitzer and Pereira, 2007; Jia and Liang, 2017; Brun and Nikoulina, 2018). This drop in performance when changing from domain Ds to domain Dt can be due to a variety of causes. It could be because of Co-variate Shift (Shi- modaira, 2000; Storkey, 2009), where the input distribution changes, but the conditional distribu- tion does not, i.e. PDs(y|x) = PDt(y|x) but PDs(x) ̸= PDt(x); or due to Concept Shift, when PDs(y|x) ̸= PDt(y|x) and PDs(x) = PDt(x), Value�of�the�Domain-shift Detection�Metric�on�Dt Expected�Accuracy Drop�on�Dt� Figure 1: In this paper we introduce several domain- shift detection metrics (x-axis) and employ them to es- timate the performance drop on a new target domain Dt by regressing on those metrics and their associated real performance drop (green dots). or Label Shift, when PDs(y) ̸= PDt(y) and PDs(x|y) = PDt(x|y) (Zhang et al., 2013; Lip- ton et al., 2018), or a mix between them (Moreno- Torres et al., 2012; Quionero-Candela et al., 2009). Such changes are often the norm in many real- world applications, creating an urge in the industry to continuously check that models in production are performing well in the wild or when there is a new client or a new type of data. However, con- tinuously sampling and annotating data-points can be prohibitively slow and costly, particularly when a large annotated sample is needed to correctly represent the target joint distribution or when the change can be gradual …

Content truncated to 500 words