Online Updating of Word Representations for Part-of-Speech Tagging

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1329–1334, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Online Updating of Word Representations for Part-of-Speech Tagging Wenpeng Yin LMU Munich wenpeng@cis.lmu.de Tobias Schnabel Cornell University tbs49@cornell.edu Hinrich Sch¨utze LMU Munich inquiries@cislmu.org Abstract We propose online unsupervised domain adaptation (DA), which is performed in- crementally as data comes in and is appli- cable when batch DA is not possible. In a part-of-speech (POS) tagging evaluation, we find that online unsupervised DA per- forms as well as batch DA. 1 Introduction Unsupervised domain adaptation is a scenario that practitioners often face when having to build ro- bust NLP systems. They have labeled data in the source domain, but wish to improve performance in the target domain by making use of unlabeled data alone. Most work on unsupervised domain adaptation in NLP uses batch learning: It assumes that a large corpus of unlabeled data of the tar- get domain is available before testing. However, batch learning is not possible in many real-world scenarios where incoming data from a new target domain must be processed immediately. More im- portantly, in many real-world scenarios the data does not come with neat domain labels and it may not be immediately obvious that an input stream is suddenly delivering data from a new domain. Consider an NLP system that analyzes emails at an enterprise. There is a constant stream of in- coming emails and it changes over time – without any clear indication that the models in use should be adapted to the new data distribution. Because the system needs to work in real-time, it is also de- sirable to do any adaptation of the system online, without the need of stopping the system, changing it and restarting it as is done in batch mode. In this paper, we propose online unsupervised domain adaptation as an extension to traditional unsupervised DA. In online unsupervised DA, do- main adaptation is performed incrementally as data comes in. Specifically, we adopt a form of representation learning. In our experiments, the incremental updating will be performed for repre- sentations of words. Each time a word is encoun- tered in the stream of data at test time, its repre- sentation is updated. To the best of our knowledge, the work re- ported here is the first study of online unsuper- vised DA. More specifically, we evaluate online unsupervised DA for the task of POS tagging. We compare POS tagging results for three distinct ap- proaches: static (the baseline), batch learning and online unsupervised DA. Our results show that online unsupervised DA is comparable in perfor- mance to batch learning while requiring no retrain- ing or prior data in the target domain. 2 Experimental setup Tagger. We reimplemented the FLORS tagger (Schnabel and Sch¨utze, 2014), a fast and simple tagger that performs well in DA. It treats POS tag- ging as a window-based (as opposed to sequence classification), multilabel classification problem. FLORS is ideally suited for online …

Content truncated to 500 words