Hierarchical Bayesian Domain Adaptation

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 602–610, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Hierarchical Bayesian Domain Adaptation Jenny Rose Finkel and Christopher D. Manning Computer Science Department Stanford University Stanford, CA 94305 {jrfinkel|manning}@cs.stanford.edu Abstract Multi-task learning is the problem of maxi- mizing the performance of a system across a number of related tasks. When applied to mul- tiple domains for the same task, it is similar to domain adaptation, but symmetric, rather than limited to improving performance on a target domain. We present a more principled, better performing model for this problem, based on the use of a hierarchical Bayesian prior. Each domain has its own domain-specific parame- ter for each feature but, rather than a constant prior over these parameters, the model instead links them via a hierarchical Bayesian global prior. This prior encourages the features to have similar weights across domains, unless there is good evidence to the contrary. We show that the method of (Daum´e III, 2007), which was presented as a simple “prepro- cessing step,” is actually equivalent, except our representation explicitly separates hyper- parameters which were tied in his work. We demonstrate that allowing different values for these hyperparameters significantly improves performance over both a strong baseline and (Daum´e III, 2007) within both a conditional random field sequence model for named en- tity recognition and a discriminatively trained dependency parser. 1 Introduction The goal of multi-task learning is to improve perfor- mance on a set of related tasks, when provided with (potentially varying quantities of) annotated data for each of the tasks. It is very closely related to domain adaptation, a far more common task in the natural language processing community, but with two pri- mary differences. Firstly, in domain adaptation the different tasks are actually just different domains. Secondly, in multi-task learning the focus is on im- proving performance across all tasks, while in do- main adaptation there is a distinction between source data and target data, and the goal is to improve per- formance on the target data. In the present work we focus on domain adaptation, but like the multi-task setting, we wish to improve performance across all domains and not a single target domains. The word domain is used here somewhat loosely: it may refer to a topical domain or to distinctions that linguists might term mode (speech versus writing) or regis- ter (formal written prose versus SMS communica- tions). For example, one may have a large amount of parsed newswire, and want to use it to augment a much smaller amount of parsed e-mail, to build a higher quality parser for e-mail data. We also con- sider the extension to the task where the annotation is not the same, but is consistent, across domains (that is, some domains may be annotated with more information than others). This problem is important because it is omni- present in real life natural language processing tasks. Annotated data is expensive to …

Content truncated to 500 words