Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog

Proceedings of NAACL-HLT 2019, pages 3795–3805 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3795 Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog Sebastian Schuster∗ Stanford Linguistics sebschu@stanford.edu Sonal Gupta Facebook Conversational AI sonalgupta@fb.com Rushin Shah Facebook Conversational AI rushinshah@fb.com Mike Lewis Facebook AI Research mikelewis@fb.com Abstract One of the first steps in the utterance interpre- tation pipeline of many task-oriented conver- sational AI systems is to identify user intents and the corresponding slots. Since data col- lection for machine learning models for this task is time-consuming, it is desirable to make use of existing data in a high-resource lan- guage to train models in low-resource lan- guages. However, development of such mod- els has largely been hindered by the lack of multilingual training data. In this paper, we present a new data set of 57k annotated ut- terances in English (43k), Spanish (8.6k) and Thai (5k) across the domains weather, alarm, and reminder. We use this data set to evalu- ate three different cross-lingual transfer meth- ods: (1) translating the training data, (2) us- ing cross-lingual pre-trained embeddings, and (3) a novel method of using a multilingual ma- chine translation encoder as contextual word representations. We find that given several hundred training examples in the the target language, the latter two methods outperform translating the training data. Further, in very low-resource settings, multilingual contextual word representations give better results than using cross-lingual static embeddings. We also compare the cross-lingual methods to us- ing monolingual resources in the form of con- textual ELMo representations and find that given just small amounts of target language data, this method outperforms all cross-lingual methods, which highlights the need for more sophisticated cross-lingual methods. 1 Introduction One of the first steps in many conversational AI systems that are used to parse utterances in per- sonal assistants is the identification of what the user intends to do (the intent) as well as the argu- ments of the intent (the slots) (Mesnil et al., 2013; ∗ Work carried out during an internship at Facebook. Liu and Lane, 2016). For example, for a request such as Set an alarm for tomorrow at 7am, a first step in fulfilling such a request is to identify that the user’s intent is to set an alarm and that the re- quired time argument of the request is expressed by the phrase tomorrow at 7am. Given these properties of the task, the problem can be stated as a joint sentence classification (for intent classification) and sequence labeling (for slot detection) task and therefore naturally lends itself to using a biLSTM-CRF sequence labeling model (Lample et al., 2016; Vu, 2016) where the biLSTM layer is also used as the input for a pro- jection layer for intent detection. These models are very powerful and given enough training data, they achieve very high ac- curacy on the intent classification as well as the slot detection task. However, given the require- ment of large amounts of labeled training data, …

Content truncated to 500 words