Zipfian corruptions for robust POS tagging

Proceedings of NAACL-HLT 2013, pages 668–672, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Zipfian corruptions for robust POS tagging Anders Søgaard Center for Language Technology University of Copenhagen soegaard@hum.ku.dk Abstract Inspired by robust generalization and adver- sarial learning we describe a novel approach to learning structured perceptrons for part-of- speech (POS) tagging that is less sensitive to domain shifts. The objective of our method is to minimize average loss under random distri- bution shifts. We restrict the possible target distributions to mixtures of the source distri- bution and random Zipfian distributions. Our algorithm is used for POS tagging and eval- uated on the English Web Treebank and the Danish Dependency Treebank with an average 4.4% error reduction in tagging accuracy. 1 Introduction Supervised learning approaches have advanced the state of the art on a variety of tasks in natural lan- guage processing, often resulting in systems ap- proaching the level of inter-annotator agreement on in-domain data, e.g. in POS tagging, where Shen et al. (2007) report a tagging accuracy of 97.3%. However, performance of state-of-the-art supervised systems is known to drop considerably on out-of- domain data. State-of-the-art POS taggers trained on the Penn Treebank (Marcus et al., 1993) mapped to Google’s universal tag set (Petrov et al., 2011) achieve tagging accuracies in the range of 89–91% on Web 2.0 data (Petrov and McDonald, 2012) . To bridge this gap we may consider using semi- supervised or transfer learning methods to adjust to new target domains (Blitzer et al., 2006; Daume III, 2007), pooling unlabeled data from those domains. However, in many applications this is not possible. If we want to provide an online service or design a piece of software with many potential users covering a wide range of use cases, we do not know the target domain in advance. This is the usual problem of ro- bust learning, but in this paper we describe a novel learning algorithm that goes beyond robust learning by making various assumptions about the difference between the source domain and the (unknown) target domain. Under these assumptions we can minimize average loss under (all possible or a representative sample of) domain shifts. We evaluate our approach on two recently introduced cross-domain POS tag- ging datasets. Our approach is inspired by work in robust gen- eralization (Ben-Tal and Nemirovski, 1998; Trafalis and Gilbert, 2007) and adversarial learning (Glober- son and Roweis, 2006; Dekel and Shamir, 2008; Søgaard and Johannsen, 2012). Our approach also bears similarities to feature bagging (Sutton et al., 2006). Sutton et al. (2006) noted that in learning of linear models useful features are often swamped by correlating, but more indicative features. If the more indicative features are absent in the target domain due to out-of-vocabulary (OOV) effects, we are left with the swamped features which were not updated properly. This is, indirectly, the problem solved in adversarial learning with corrupted data points. Ad- versarial learning can also be seen as a way of av- eraging exponentially many models (Hinton …

Content truncated to 500 words