The Necessity of Combining Adaptation Methods

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 767–777, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics The Necessity of Combining Adaptation Methods Ming-Wei Chang, Michael Connor and Dan Roth University of Illinois at Urbana Champaign Urbana, IL 61801 {mchang21,connor2,danr}@uiuc.edu Abstract Problems stemming from domain adaptation continue to plague the statistical natural lan- guage processing community. There has been continuing work trying to find general purpose algorithms to alleviate this problem. In this paper we argue that existing general purpose approaches usually only focus on one of two issues related to the difficulties faced by adap- tation: 1) difference in base feature statistics or 2) task differences that can be detected with labeled data. We argue that it is necessary to combine these two classes of adaptation algorithms, using evidence collected through theoretical analy- sis and simulated and real-world data exper- iments. We find that the combined approach often outperforms the individual adaptation approaches. By combining simple approaches from each class of adaptation algorithm, we achieve state-of-the-art results for both Named Entity Recognition adaptation task and the Preposition Sense Disambiguation adaptation task. Second, we also show that applying an adaptation algorithm that finds shared repre- sentation between domains often impacts the choice in adaptation algorithm that makes use of target labeled data. 1 Introduction While recent advances in statistical modeling for natural language processing are exciting, the prob- lem of domain adaptation remains a big challenge. It is widely known that a classifier trained on one do- main (e.g. news domain) usually performs poorly on a different domain (e.g. medical domain) (Jiang and Zhai, 2007; Daum´e III, 2007). The inability of cur- rent statistical models to handle multiple domains is one of the key obstacles hindering the progress of NLP. Several general purpose algorithms have been proposed to address the domain adaptation prob- lem: (Blitzer et al., 2006; Jiang and Zhai, 2007; Daum´e III, 2007; Finkel and Manning, 2009). It is widely believed that the drop in performance of statistical models on new domains is due to the shift of the joint distribution of labels and examples, P(Y, X), from domain to domain, where X repre- sents the input space and Y represents the output space. In general, we can separate existing adap- tation algorithms into two categories: Focuses on P(X) This type of adaptation algo- rithm attempts to resolve the difference between the feature space statistics of two domains. While many different techniques have been proposed, the com- mon goal of these algorithms is to find a better shared representation that brings the source domain and the target domain closer. Often these algorithms do not use labeled examples in the target domain. The works (Blitzer et al., 2006; Huang and Yates, 2009) all belong to this category. Focuses on P(Y |X) These adaptation algorithms assume that there exists a small amount of labeled data for the target domain. Instead of training two weight vectors independently (one for source and the other …

Content truncated to 500 words