Semi-supervised Representation Learning for Domain Adaptation using Dynamic Dependency Networks

Proceedings of COLING 2012: Technical Papers, pages 2867–2882, COLING 2012, Mumbai, December 2012. Semi-supervised Representation Learning for Domain Adaptation using Dynamic Dependency Networks Min X iao Yuhong Guo Alexander Yates Department of Computer and Information Sciences Temple University, Philadelphia, PA, USA {minxiao, yuhong, yates}@temple.edu ABSTRACT Recently, various unsupervised representation learning approaches have been investigated to produce augmenting features for natural language processing systems in the open-domain learning scenarios. In this paper, we propose a dynamic dependency network model to conduct semi-supervised representation learning. It exploits existing task-specific labels in the source domain in addition to the large amount of unlabeled data from both the source and target domains to produce informative features for NLP tasks. We empirically evaluate the proposed learning technique on the part-of-speech tagging task using Wall Street Journal and MEDLINE sentences and on the syntactic chunking task using Wall Street Journal corpus and Brown corpus. Our experimental results show that the proposed semi-supervised learning model can produce more effective features than unsupervised representation learning methods for open- domain part-of-speech taggers and syntactic chunkers. KEYWORDS: Domain Adaptation, Representation Learning, POS Tagging, Syntactic Chunking. 2867 1 Introduction Existing supervised natural language processing (NLP) systems are highly domain-dependent, whose performance degrades significantly when tested on a new domain. Previous works in a variety of NLP tasks, like part-of-speech (POS) tagging (Blitzer et al., 2006; Huang and Yates, 2010; Blitzer et al., 2011), syntactic chunking (Huang and Yates, 2009; Carreras and Màrquez, 2005), named entity recognition (NER) (Daumé III, 2007; Turian et al., 2010; Daumé III and Marcu, 2006), or parsing (Sekine, 1997; McClosky et al., 2010) show that the performance of supervised NLP systems drops a lot on domains whose vocabulary differs from the vocabulary of the training data. The major reason that causes the increasing of test error on out-of-domain texts is the tradi- tional representation used in the supervised NLP systems. Most NLP systems use the lexical features for predictions. Though it works very well for various in-domain NLP tasks, they per- form poorly when tested on a different domain. There are two main reasons. First, the source and target domains may have very different vocabularies, thus some test words may never appear during the training phase. For example, “sequencing”, “metastases” and “genomic” show up frequently as lexical features in biomedical text but rarely in newswire articles. A classifier trained on newswire data thus will have seen few training examples related to sen- tences with lexical features “sequencing”, “metastases” and “genomic” (Ben-David et al., 2010, 2007). Second, the prediction function based on lexical features may change across domains. For example, “signaling” appears in “signaling that ...” from a Wall Street Journal (WSJ) arti- cle primarily as a present participle (VBG) (Marcus et al., 1993), but predominantly as a noun in “signaling pathway” from a MEDLINE text (PennBioIE, 2005). Recently, various unsupervised representation learning techniques are proposed to induce gen- eralizable latent features across domains by exploiting large amount of unlabeled data from both the source and the target …

Content truncated to 500 words