Biased Representation Learning for Domain Adaptation

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1313–1323, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Biased Representation Learning for Domain Adaptation Fei Huang , Alexander Yates Temple University Computer and Information Sciences 324 Wachman Hall Philadelphia, PA 19122 {fhuang,yates}@temple.edu Abstract Representation learning is a promising tech- nique for discovering features that allow su- pervised classifiers to generalize from a source domain dataset to arbitrary new domains. We present a novel, formal statement of the rep- resentation learning task. We argue that be- cause the task is computationally intractable in general, it is important for a representa- tion learner to be able to incorporate expert knowledge during its search for helpful fea- tures. Leveraging the Posterior Regularization framework, we develop an architecture for in- corporating biases into representation learn- ing. We investigate three types of biases, and experiments on two domain adaptation tasks show that our biased learners identify signif- icantly better sets of features than unbiased learners, resulting in a relative reduction in er- ror of more than 16% for both tasks, with re- spect to existing state-of-the-art representation learning techniques. 1 Introduction Supervised natural language processing (NLP) sys- tems have been widely used and have achieved im- pressive performance on many NLP tasks. Howev- er, they exhibit a significant drop-off in performance when tested on domains that differ from their train- ing domains. (Gildea, 2001; Sekine, 1997; Pradhan et al., 2007) One major cause for poor performance on out of-domain texts is the traditional representa- tion used by supervised NLP systems (Ben-David et al., 2007). Most systems depend on lexical features, which can differ greatly between domains, so that important words in the test data may never be seen in the training data. The connection between word- s and labels may also change across domains. For instance, “signaling” appears only as a present par- ticiple (VBG) in WSJ text (as in, “signaling that...”), but predominantly as a noun (as in “signaling path- way”) in biomedical text. Recently, several authors have found that learning new features based on distributional similarity can significantly improve domain adaptation (Blitzer et al., 2006; Huang and Yates, 2009; Turian et al., 2010; Dhillon et al., 2011). This framework is at- tractive for several reasons: experimentally, learned features can yield significant improvements over s- tandard supervised models on out-of-domain test- s. Moreover, since the representation-learning tech- niques are unsupervised, they can easily be applied to arbitrary new domains. There is no need to supply additional labeled examples for each new domain. Traditional representations still hold one signif- icant advantage over representation-learning, how- ever: because features are hand-crafted, these rep- resentations can readily incorporate the linguistic or domain expert knowledge that leads to state-of- the-art in-domain performance. In contrast, the on- ly guide for existing representation-learning tech- niques is a corpus of unlabeled text. To address this shortcoming, we introduce representation-learning techniques that incorporate a domain expert’s preferences over the learned …

Content truncated to 500 words