Importance weighting and unsupervised domain adaptation of POS taggers: a negative result

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 968–973, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Importance weighting and unsupervised domain adaptation of POS taggers: a negative result Barbara Plank, Anders Johannsen and Anders Søgaard Center for Language Technology University of Copenhagen, Denmark Njalsgade 140, DK-2300 Copenhagen S bplank@cst.dk,ajohannsen@hum.ku.dk,soegaard@hum.ku.dk Abstract Importance weighting is a generalization of various statistical bias correction tech- niques. While our labeled data in NLP is heavily biased, importance weighting has seen only few applications in NLP, most of them relying on a small amount of labeled target data. The publication bias toward reporting positive results makes it hard to say whether researchers have tried. This paper presents a negative result on unsu- pervised domain adaptation for POS tag- ging. In this setup, we only have unlabeled data and thus only indirect access to the bias in emission and transition probabili- ties. Moreover, most errors in POS tag- ging are due to unseen words, and there, importance weighting cannot help. We present experiments with a wide variety of weight functions, quantilizations, as well as with randomly generated weights, to support these claims. 1 Introduction Many NLP tasks rely on the availability of anno- tated data. The majority of annotated data, how- ever, is sampled from newswire corpora. The performance of NLP systems, e.g., part-of-speech (POS) tagger, parsers, relation extraction sys- tems, etc., drops significantly when they are ap- plied to data that departs from newswire conven- tions. So while we can extract information, trans- late and summarize newswire in major languages with some success, we are much less successful processing microblogs, chat, weblogs, answers, emails or literature in a robust way. The main rea- sons for the drops in accuracy have been attributed to factors such as previously unseen words and bi- grams, missing punctuation and capitalization, as well as differences in the marginal distribution of data (Blitzer et al., 2006; McClosky et al., 2008; Søgaard and Haulrich, 2011). The move from one domain to another (from a source to a new target domain), say from newspa- per articles to weblogs, results in a sample selec- tion bias. Our training data is now biased, since it is sampled from a related, but nevertheless dif- ferent distribution. The problem of automatically adjusting the model induced from source to a dif- ferent target is referred to as domain adaptation. Some researchers have studied domain adap- tation scenarios, where small samples of labeled data have been assumed to be available for the target domains. This is usually an unrealistic as- sumption, since even for major languages, small samples are only available from a limited number of domains, and in this work we focus on unsuper- vised domain adaptation, assuming only unlabeled target data is available. Jiang and Zhai (2007), Foster et al. (2010; Plank and Moschitti (2013) and Søgaard and Haulrich (2011) have previously tried to use importance weighting to correct sample bias in NLP. Im- portance weighting means assigning a …

Content truncated to 500 words