Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3226–3237 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3226 Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding Yingmei Guo1∗ Linjun Shou2 Jian Pei3 Ming Gong2 Mingxing Xu1 Zhiyong Wu1 Daxin Jiang2† 1Department of Computer Science and Technology, Tsinghua University 2NLP Group, Microsoft STCA 3School of Computing Science, Simon Fraser University {guoym18@mails.,xumx@,zywu@sz.}tsinghua.edu.cn {lisho,migon,djiang}@microsoft.com jpei@cs.sfu.ca Abstract Lack of training data presents a grand chal- lenge to scaling out spoken language under- standing (SLU) to low-resource languages. Al- though various data augmentation approaches have been proposed to synthesize training data in low-resource target languages, the aug- mented data sets are often noisy, and thus im- pede the performance of SLU models. In this paper we focus on mitigating noise in aug- mented data. We develop a denoising training approach. Multiple models are trained with data produced by various augmented methods. Those models provide supervision signals to each other. The experimental results show that our method outperforms the existing state of the art by 3.05 and 4.24 percentage points on two benchmark datasets, respectively. The code will be made open sourced on github. 1 Introduction Spoken language understanding (SLU) is a key component in task-oriented dialogue systems. SLU consists of two subtasks: intent detection and slot tagging (Wang et al., 2005; Tur and De Mori, 2011). Although promising progress has been achieved on SLU in English (Liu and Lane, 2016; Peng et al., 2020; Huang et al., 2020), those methods need large amounts of training data, and thus cannot be applied to low-resource languages where zero or few training data is available. In this paper, we target at the extreme setting for cross-lingual SLU where no labeled data in target languages is assumed, which is critical for industry practice, since annotating a large SLU dataset with high quality for every language is simply infeasible. Existing cross-lingual transfer learning methods mainly build on pre-trained cross-lingual word em- beddings (Ruder et al., 2019) or contextual mod- els (Wu and Dredze, 2019; Huang et al., 2019; *Work is done during internship at NLP Group, Microsoft STCA. †Corresponding author. Lample and Conneau, 2019; Conneau et al., 2020), which represent texts with similar meaning in dif- ferent languages close to each other in a shared vector space. Those approaches often show good performance on intent detection. The results on slot tagging, however, are often unsatisfactory, es- pecially for distant languages, which are dramati- cally different from English in scripts, morphology, or syntax (Upadhyay et al., 2018; Schuster et al., 2019; Li et al., 2020a). Several studies (Schuster et al., 2019; Liu et al., 2020b; Li et al., 2020a) show that adding trans- lated data into the fine-tuning process of pretrained models can improve the results of cross-lingual SLU substantially when no golden-labeled train- ing data in target languages is available. For ex- ample, machine translation can be employed to translate the training data in English into target languages, and some …

Content truncated to 500 words