We’re Not in Kansas Anymore: Detecting Domain Changes in Streams

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 585–595, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics We’re Not in Kansas Anymore: Detecting Domain Changes in Streams †⋆Mark Dredze and †∗Tim Oates and †‡Christine Piatko †Human Language Technology Center of Excellence, ⋆Center for Language and Speech Processing, ‡Applied Physics Lab Johns Hopkins University ∗University of Maryland, Baltimore County mdredze@cs.jhu.edu, oates@umbc.edu, christine.piatko@jhuapl.edu Abstract Domain adaptation, the problem of adapting a natural language processing system trained in one domain to perform well in a differ- ent domain, has received significant attention. This paper addresses an important problem for deployed systems that has received little at- tention – detecting when such adaptation is needed by a system operating in the wild, i.e., performing classification over a stream of unlabeled examples. Our method uses A- distance, a metric for detecting shifts in data streams, combined with classification margins to detect domain shifts. We empirically show effective domain shift detection on a variety of data sets and shift conditions. 1 Introduction Consider a named entity recognition system trained on newswire stories. Given annotated documents containing sentences like “Tony Hayward has faced fresh criticism for taking time off to go sailing . . .” we would like to learn a model that will allow us to recognize that “Obama” and “BP” are named enti- ties in a sentence like “Obama summoned BP ex- ecutives . . .”. When all of the documents come from one data distribution, like newswire articles, this tends to work well. However, the sentence “OBAMA SUMMONED BP EXECUTIVES . . .” from transcribed broadcast news, and others like it, will probably lead to poor results because the fea- tures it relies on have changed. For example, capi- talization patterns are no longer a good indicator of the presence of a named entity and appositives are not indicated by punctuation. This problem of do- main shift is a pervasive problem in NLP in which any kind of model – a parser, a POS tagger, a senti- ment classifier – is tested on data that do not match the training data. Given a model and a stream of unlabeled in- stances, we are interested in automatically detecting changes in the feature distribution that negatively impact classification accuracy. For example, a senti- ment classification model trained on book reviews may heavily weight n-grams features like “uplift- ing” and “page turner”. Those features may never occur in reviews of kitchen appliances that get mixed in at test time, and useful features in this new do- main like “efficient” and “noisy compressor” will have never been seen during training and therefore not be in the model. Furthermore, we do not assume labeled instances are available to help detect these harmful changes. Other tasks related to changes in data distributions, like detecting concept drift in which the labeling function changes, may require la- beled instances, but that is not the focus of this paper. There is significant work on the …

Content truncated to 500 words