Domain Adaptation with Structural Correspondence Learning

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 120–128, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Domain Adaptation with Structural Correspondence Learning John Blitzer Ryan McDonald Fernando Pereira {blitzer|ryantm|pereira}@cis.upenn.edu Department of Computer and Information Science, University of Pennsylvania 3330 Walnut Street, Philadelphia, PA 19104, USA Abstract Discriminative learning methods are widely used in natural language process- ing. These methods work best when their training and test data are drawn from the same distribution. For many NLP tasks, however, we are confronted with new domains in which labeled data is scarce or non-existent. In such cases, we seek to adapt existing models from a resource- rich source domain to a resource-poor target domain. We introduce structural correspondence learning to automatically induce correspondences among features from different domains. We test our tech- nique on part of speech tagging and show performance gains for varying amounts of source and target training data, as well as improvements in target domain parsing accuracy using our improved tagger. 1 Introduction Discriminative learning methods are ubiquitous in natural language processing. Discriminative tag- gers and chunkers have been the state-of-the-art for more than a decade (Ratnaparkhi, 1996; Sha and Pereira, 2003). Furthermore, end-to-end sys- tems like speech recognizers (Roark et al., 2004) and automatic translators (Och, 2003) use increas- ingly sophisticated discriminative models, which generalize well to new data that is drawn from the same distribution as the training data. However, in many situations we may have a source domain with plentiful labeled training data, but we need to process material from a target do- main with a different distribution from the source domain and no labeled data. In such cases, we must take steps to adapt a model trained on the source domain for use in the target domain (Roark and Bacchiani, 2003; Florian et al., 2004; Chelba and Acero, 2004; Ando, 2004; Lease and Char- niak, 2005; Daum´e III and Marcu, 2006). This work focuses on using unlabeled data from both the source and target domains to learn a common feature representation that is meaningful across both domains. We hypothesize that a discrimi- native model trained in the source domain using this common feature representation will general- ize better to the target domain. This representation is learned using a method we call structural correspondence learning (SCL). The key idea of SCL is to identify correspon- dences among features from different domains by modeling their correlations with pivot features. Pivot features are features which behave in the same way for discriminative learning in both do- mains. Non-pivot features from different domains which are correlated with many of the same pivot features are assumed to correspond, and we treat them similarly in a discriminative learner. Even on the unlabeled data, the co-occurrence statistics of pivot and non-pivot features are likely to be sparse, and we must model them in a com- pact way. There are many choices for modeling co-occurrence data (Brown et al., 1992; Pereira et al., 1993; Blei …

Content truncated to 500 words