A General Regularization Framework for Domain Adaptation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 950–954, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics A General Regularization Framework for Domain Adaptation Wei Lu1 and Hai Leong Chieu2 and Jonathan L¨ofgren3 1Singapore University of Technology and Design 2DSO National Laboratories 3Uppsala University luwei@sutd.edu.sg, chaileon@dso.org.sg, lofgren021@gmail.com Abstract We propose a domain adaptation framework, and formally prove that it generalizes the fea- ture augmentation technique in (Daum´e III, 2007) and the multi-task regularization frame- work in (Evgeniou and Pontil, 2004). We show that our framework is strictly more gen- eral than these approaches and allows practi- tioners to tune hyper-parameters to encourage transfer between close domains and avoid neg- ative transfer between distant ones. 1 Introduction Domain adaptation (DA) is an important problem that has received substantial attention in natural lan- guage processing (Blitzer et al., 2006; Daum´e III, 2007; Finkel and Manning, 2009; Daum´e III et al., 2010). In this paper, we propose a novel regular- ization framework which allows DA practitioners to tune hyper-parameters to encourage transfer be- tween close domains, and avoid negative transfer (Rosenstein et al., 2005) between distant ones. In our framework, model parameters in multiple do- mains are learned jointly and constrained to remain close to one another. In the transfer learning tax- onomy (Pan and Yang, 2010), our framework falls under the parameter-transfer category for multi-task inductive learning. We show that our framework generalizes the frustratingly easy domain adapta- tion (FEDA) in Daum´e III (2007), Finkel and Man- ning (2009), and the regularised multi-task learning of Evgeniou and Pontil (2004). At the same time, it provides us with hyper-parameters to control the amount of transfer between domains. 2 Domain Adaptation Framework Given labeled data from N domains, D1, . . . , DN, traditional machine learning maximizes the follow- ing objective function for each domain Di: O(Di; wi) = Li(Di; wi) − λi||wi||2, (1) and we maximize Li by tuning the parameter vector wi. For example, Li can be the log-likelihood or the negative hinge loss. The term λi||wi||2 is the L2- regularization term where λi is a positive scalar. In our framework, we propose to maximize N � i=1 Li(Di; wi) − N � i=1 η0,i||wi||2 − � 1≤j<k≤N ηj,k||wj − wk||2, (2) where ηj,k are parameters controlling the transfer between domains. In the next sections, we show how our framework generalizes existing works. 2.1 Frustratingly Easy DA The FEDA approach was introduced by Daum´e III (2007) and later formalized by Finkel and Man- ning (2009) within a hierarchical Bayesian DA framework. While simple, the approach has often been shown to be effective. In this section, we show that our framework generalizes the FEDA approach. The FEDA approach defines a new augmented feature space by duplicating each feature in Di to a “general” domain. Therefore each parameter in wi has a corresponding parameter in w0, and: L′ i(Di; wi, w0) = Li(Di; wi + w0) (3) This directly leads to the following remark: 950 Remark …

Content truncated to 500 words