Domain Attention with an Ensemble of Experts

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 643–653 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1060 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 643–653 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1060 Domain Attention with an Ensemble of Experts Young-Bum Kim† Karl Stratos‡ Dongchan Kim† †Microsoft AI and Research ‡Bloomberg L. P. {ybkim, dongchan.kim}@microsoft.com me@karlstratos.com Abstract An important problem in domain adapta- tion is to quickly generalize to a new do- main with limited supervision given K ex- isting domains. One approach is to re- train a global model across all K + 1 do- mains using standard techniques, for in- stance Daum´e III (2009). However, it is desirable to adapt without having to re- estimate a global model from scratch each time a new domain with potentially new intents and slots is added. We describe a solution based on attending an ensemble of domain experts. We assume K domain- specific intent and slot models trained on respective domains. When given domain K + 1, our model uses a weighted combi- nation of the K domain experts’ feedback along with its own opinion to make predic- tions on the new domain. In experiments, the model significantly outperforms base- lines that do not use domain adaptation and also performs better than the full re- training approach. 1 Introduction An important problem in domain adaptation is to quickly generalize to a new domain with limited supervision given K existing domains. In spo- ken language understanding, new domains of in- terest for categorizing user utterances are added on a regular basis1. For instance, we may 1A scenario frequently arising in practice is having a re- quest for creating a new virtual domain targeting a specific application. One typical use case is that of building natural language capability through intent and slot modeling (with- out actually building a domain classifier) targeting a specific application. add ORDERPIZZA domain and desire a domain- specific intent and semantic slot tagger with a lim- ited amount of training data. Training only on the target domain fails to utilize the existing resources in other domains that are relevant (e.g., labeled data for PLACES domain with place name, location as the slot types), but naively training on the union of all domains does not work well since different domains can have widely varying distributions. Domain adaptation offers a balance between these extremes by using all data but simultane- ously distinguishing domain types. A common approach for adapting to a new domain is to re- train a global model across all K + 1 domains us- ing well-known techniques, for example the fea- ture augmentation method of Daum´e III (2009) which trains a single model that has one domain- invariant component along with K + 1 domain- specific components each of which is specialized in a particular domain. While such a global model is effective, it requires …

Content truncated to 500 words