Frustratingly Easy Neural Domain Adaptation

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 387–396, Osaka, Japan, December 11-17 2016. Frustratingly Easy Neural Domain Adaptation Young-Bum Kim Microsoft Redmond, WA ybkim@microsoft.com Karl Stratos∗ Bloomberg L. P. New York, NY me@karlstratos.com Ruhi Sarikaya† Amazon Seattle, WA rsarikay@amazon.com Abstract Popular techniques for domain adaptation such as the feature augmentation method of Daum´e III (2009) have mostly been considered for sparse binary-valued features, but not for dense real- valued features such as those used in neural networks. In this paper, we describe simple neural extensions of these techniques. First, we propose a natural generalization of the feature augmen- tation method that uses K + 1 LSTMs where one model captures global patterns across all K domains and the remaining K models capture domain-specific information. Second, we propose a novel application of the framework for learning shared structures by Ando and Zhang (2005) to domain adaptation, and also provide a neural extension of their approach. In experiments on slot tagging over 17 domains, our methods give clear performance improvement over Daum´e III (2009) applied on feature-rich CRFs. 1 Introduction There are often multiple types of data that share common characteristics. For example, in this work we are interested in slot tagging of user queries under as many as 17 different domains. Correctly labeling a given query requires the knowledge of the domain that the query is in. A word such as “home” can be labeled as either contact name under the COMMUNICATION domain, place type under the PLACES domain, or home screen under the XBOXENTERTAINMENTSEARCH domain. On the other hand, a function word such as “the” is labeled as others across all domains. Ideally, we would like to train on all sources of data in order to learn about words that are shared across domains, but naively combining these sources for training does not work well since this ignores label inconsistencies across domains. The other extreme is to only use data only from the domain of interest, but it misses out opportunities to learn global patterns shared by multiple domains. The goal of domain adaptation is precisely to address this conundrum: how can we learn from multiple sources of data but retain the integrity of individual domains? There has been much progress in domain adaptation. A notable example is the feature augmentation method of Daum´e III (2009), whose key insight is that if we partition the model parameters to those that handle common patterns and those that handle domain-specific patterns, the model is forced to learn from all domains yet preserve domain-specific knowledge. The method of Daum´e III (2009) is usually considered for sparse binary-valued features which underlie conventional NLP systems. With such features, the parameter partitioning can be achieved with trivial data preprocessing: by conjoining feature types with domain indicators and using them alongside the original feature types. But it is not clear how this approach can be extended to dense real-valued feature values, which are used in many recent NLP systems based …

Content truncated to 500 words