Cross-lingual Transfer Learning with Data Selection for Large-Scale Spoken Language Understanding

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1455–1460, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1455 Cross-lingual Transfer Learning with Data Selection for Large-Scale Spoken Language Understanding Quynh Do Amazon Aachen, Germany doquynh@amazon.com Judith Gaspers Amazon Aachen, Germany gaspers@amazon.de Abstract A typical cross-lingual transfer learning ap- proach boosting model performance on a resource-poor language is to pre-train the model on all available supervised data from another resource-rich language. However, in large-scale systems, this leads to high training times and computational requirements. In ad- dition, characteristic differences between the source and target languages raise a natural question of whether source-language data se- lection can improve the knowledge transfer. In this paper, we address this question and pro- pose a simple but effective language model based source-language data selection method for cross-lingual transfer learning in large- scale spoken language understanding. The experimental results show that with data se- lection i) the source data amount and hence training speed is reduced significantly and ii) model performance is improved. 1 Introduction Spoken Language Understanding (SLU) plays an important role in spoken language technology and is typically divided into two sub-tasks: Intent Clas- sification (IC) and Slot Filling (SF). While the for- mer identifies a speaker’s intent, the latter extracts semantic constituents from the natural language query. Recently, there have been emerging efforts on Cross-Lingual Transfer Learning (CLTL) meth- ods to reduce data requirements in deep neural net- work (DNN) based SLU. A typical approach is to pre-train the model on labeled data from a richly re- sourced language, and then either apply it directly on a target language (Upadhyay et al., 2018) or fine- tune it on a smaller amount of supervised data from a target language (Do and Gaspers, 2019). How- ever, both in SLU and other NLP tasks, prior work on CLTL typically utilized all available source data to transfer knowledge, as it has been mainly inves- tigated in rather small Academic settings. How- ever, in large-scale settings with millions of utter- ances, this would lead to costly training times, high computational requirements and optimization dif- ficulties. Moreover, the different characteristics between the source and target languages raise a natural question of whether source-language data selection in which only the most relevant source instances are picked for pre-training can improve CLTL performance, as a considerable amount of source utterances might be “irrelevant” to the target language or even yield negative transfer. Addressing these questions, in this paper we ex- plore source-language data selection for CLTL in SLU, focusing especially on large-scale settings in which we assume the existence of a large amount (millions) of source data and a moderate amount (thousands) of target data. Since the effectiveness of pre-training in CLTL depends on the similarity of the source data distribution and the real distri- bution of the target language, we propose a source- language data selection method which computes the relevance score …

Content truncated to 500 words