Domain adaptation for part-of-speech tagging of noisy user-generated text

Proceedings of NAACL-HLT 2019, pages 3415–3420 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3415 Domain adaptation for part-of-speech tagging of noisy user-generated text Luisa M¨arz and Dietrich Trautmann and Benjamin Roth CIS, University of Munich (LMU) Munich, Germany {luisa.maerz, dietrich, beroth}@cis.lmu.de Abstract The performance of a Part-of-speech (POS) tagger is highly dependent on the domain of the processed text, and for many domains there is no or only very little training data avail- able. This work addresses the problem of POS tagging noisy user-generated text using a neural network. We propose an architec- ture that trains an out-of-domain model on a large newswire corpus, and transfers those weights by using them as a prior for a model trained on the target domain (a data-set of Ger- man Tweets) for which there is very little an- notations available. The neural network has two standard bidirectional LSTMs at its core. However, we find it crucial to also encode a set of task-specific features, and to obtain reliable (source-domain and target-domain) word rep- resentations. Experiments with different reg- ularization techniques such as early stopping, dropout and fine-tuning the domain adaptation prior weights are conducted. Our best model uses external weights from the out-of-domain model, as well as feature embeddings, pre- trained word and sub-word embeddings and achieves a tagging accuracy of slightly over 90%, improving on the previous state of the art for this task. 1 Introduction Part-of-speech (POS) tagging is a prerequisite for many applications and necessary for a wide range of tools for computational linguists. The state- of-the art method to implement a tagger is to use neural networks (Ma and Hovy, 2016; Yang et al., 2018). The performance of a POS tagger is highly dependent on the domain of the processed text and the availability of sufficient training data (Schn- abel and Sch¨utze, 2014). Existing POS taggers for canonical German text already achieve very good results around 97% accuracy, e.g. (Schmid, 1999; Plank et al., 2016). When applying these trained models to out-of-domain data the performance de- creases drastically. One of the domains where there is not enough data is online conversational text in platforms such as Twitter, where the very informal language ex- hibits many phenomena that differ significantly from canonical written language. In this work, we propose a neural network that combines a character-based encoder and embed- dings of features from previous non-neural ap- proaches (that can be interpreted as an inductive bias to guide the learning task). We further show that the performance of this already effective tag- ger can be improved significantly by incorporating external weights using a mechanism of domain- specific L2-regularization during the training on in-domain data. This approach establishes state- of-the-art results of 90.3% accuracy on the Ger- man Twitter corpus of Rehbein (2013). 2 Related Work The first POS tagging approach for German Twit- ter data was conducted by Rehbein (2013) and reaches an accuracy of 88.8% on the test set using a CRF. They use …

Content truncated to 500 words