New Transfer Learning Techniques for Disparate Label Sets

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 473–482, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics New Transfer Learning Techniques for Disparate Label Sets Young-Bum Kim† Karl Stratos‡ Ruhi Sarikaya† Minwoo Jeong† †Microsoft Corporation, Redmond, WA ‡Columbia University, New York, NY {ybkim, ruhi.sarikaya, minwoo.jeong}@microsoft.com stratos@cs.columbia.edu Abstract In natural language understanding (NLU), a user utterance can be labeled differently depending on the domain or application (e.g., weather vs. calendar). Standard domain adaptation techniques are not di- rectly applicable to take advantage of the existing annotations because they assume that the label set is invariant. We propose a solution based on label embeddings in- duced from canonical correlation analysis (CCA) that reduces the problem to a stan- dard domain adaptation task and allows use of a number of transfer learning tech- niques. We also introduce a new trans- fer learning technique based on pretrain- ing of hidden-unit CRFs (HUCRFs). We perform extensive experiments on slot tag- ging on eight personal digital assistant do- mains and demonstrate that the proposed methods are superior to strong baselines. 1 Introduction The main goal of NLU is to automatically extract the meaning of spoken or typed queries. In recent years, this task has become increasingly impor- tant as more and more speech-based applications have emerged. Recent releases of personal dig- ital assistants such as Siri, Google Now, Dragon Go and Cortana in smart phones provide natu- ral language based interface for a variety of do- mains (e.g. places, weather, communications, re- minders). The NLU in these domains are based on statistical machine learned models which re- quire annotated training data. Typically each do- main has its own schema to annotate the words and queries. However the meaning of words and utter- ances could be different in each domain. For ex- ample, “sunny” is considered a weather condition in the weather domain but it may be a song title in a music domain. Thus every time a new applica- tion is developed or a new domain is built, a sig- nificant amount of resources is invested in creating annotations specific to that application or domain. One might attempt to apply existing techniques (Blitzer et al., 2006; Daum´e III, 2007) in domain adaption to this problem, but a straightforward ap- plication is not possible because these techniques assume that the label set is invariant. In this work, we provide a simple and effec- tive solution to this problem by abstracting the la- bel types using the canonical correlation analysis (CCA) by Hotelling (Hotelling, 1936) a powerful and flexible statistical technique for dimensional- ity reduction. We derive a low dimensional rep- resentation for each label type that is maximally correlated to the average context of that label via CCA. These shared label representations, or label embeddings, allow us to map label types across different domains and reduce the setting to a stan- dard domain adaptation problem. After the map- ping, we …

Content truncated to 500 words