Cross-lingual Spoken Language Understanding with Regularized Representation Alignment

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7241–7251, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7241 Cross-lingual Spoken Language Understanding with Regularized Representation Alignment Zihan Liu, Genta Indra Winata, Peng Xu, Zhaojiang Lin, Pascale Fung Center for Artificial Intelligence Research (CAiRE) Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong zihan.liu@connect.ust.hk, pascale@ece.ust.hk Abstract Despite the promising results of current cross- lingual models for spoken language under- standing systems, they still suffer from imper- fect cross-lingual representation alignments between the source and target languages, which makes the performance sub-optimal. To cope with this issue, we propose a regular- ization approach to further align word-level and sentence-level representations across lan- guages without any external resource. First, we regularize the representation of user ut- terances based on their corresponding labels. Second, we regularize the latent variable model (Liu et al., 2019a) by leveraging adver- sarial training to disentangle the latent vari- ables. Experiments on the cross-lingual spo- ken language understanding task show that our model outperforms current state-of-the-art methods in both few-shot and zero-shot scenar- ios, and our model, trained on a few-shot set- ting with only 3% of the target language train- ing data, achieves comparable performance to the supervised training with all the training data.1 1 Introduction Data-driven neural-based supervised training ap- proaches have shown effectiveness in spoken lan- guage understanding (SLU) systems (Goo et al., 2018; Chen et al., 2019; Haihong et al., 2019). However, collecting large amounts of high-quality training data is not only expensive but also time- consuming, which makes these approaches not scal- able to low-resource languages due to the scarcity of training data. Cross-lingual adaptation has natu- rally arisen to cope with this issue, which leverages the training data in rich-resource source languages and minimizes the requirement of training data in low-resource target languages. 1The code is available in https://github.com/ zliucr/crosslingual-slu. https://www.mathcha.io/editor Llr wa wb Utterance Enc. sa sb Label Enc. cos(u , u a b) l , l cos( a b) minimize utterance dist label dist Utterance Enc. Label Enc. w Utterance 휇S i Latent Va FC Slot Fil uniform one-hot adversarial training Attention Attention Attention Attention Llvm Lfc ua ub la lb Do you have thursday's weather report Cuál es el informe meteorológico mañana O O O B-time B-weather I-weather O O O B-time B-weather I-weather Utter. Utter. Intent Slots Find Weather Intent Slots Find Weather English Training sample Spanish Test sample Figure 1: Illustration of cross-lingual spoken language understanding systems, where English is the source lan- guage and Spanish is the target language. In general, there are two challenges in cross- lingual adaptation. First, the imperfect alignment of word-level representations between the source and target language limits the adaptation perfor- mance. Second, even though we assume that the word-level alignment is perfect, the sentence-level alignment is still imperfect owing to grammatical and syntactical variances across languages. There- fore, we emphasize that cross-lingual methods should focus on …

Content truncated to 500 words