Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9903 - 9918 December 7-11, 2022 ©2022 Association for Computational Linguistics Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding Shining Liang1,2,3∗, Linjun Shou3, Jian Pei4, Ming Gong3, Wanli Zuo1,2, Xianglin Zuo1,2†, Daxin Jiang3† 1College of Computer Science and Technology, Jilin University 2Key laboratory of Symbolic Computation and Knowledge Engineering, MOE 3STCA, Microsoft 4Department of Electrical & Computer Engineering, Duke University {liangsn17,zuoxl17}@mails.jlu.edu.cn; {lisho,migon,djiang}@microsoft.com; j.pei@duke.edu; zuowl@jlu.edu.cn Abstract Despite the great success of spoken language understanding (SLU) in high-resource lan- guages, it remains challenging in low-resource languages mainly due to the lack of labeled training data. The recent multilingual code- switching approach achieves better alignments of model representations across languages by constructing a mixed-language context in zero- shot cross-lingual SLU. However, current code- switching methods are limited to implicit align- ment and disregard the inherent semantic struc- ture in SLU, i.e., the hierarchical inclusion of utterances, slots, and words. In this paper, we propose to model the utterance-slot-word struc- ture by a multi-level contrastive learning frame- work at the utterance, slot, and word levels to facilitate explicit alignment. Novel code- switching schemes are introduced to gener- ate hard negative examples for our contrastive learning framework. Furthermore, we develop a label-aware joint model leveraging label se- mantics to enhance the implicit alignment and feed to contrastive learning. Our experimental results show that our proposed methods signifi- cantly improve the performance compared with the strong baselines on two zero-shot cross- lingual SLU benchmark datasets. 1 Introduction Spoken language understanding (SLU) is a criti- cal component of goal-oriented dialogue systems, which consists of two subtasks: intent detection and slot filling (Wang et al., 2005). Recently, massive efforts based on the joint training paradigm (Chen et al., 2019; Qin et al., 2021) have shown superior performance in English. However, the majority of them require large amounts of labeled training data, which limits the scalability to low-resource languages with little or no training data. Zero- shot cross-lingual approaches have arisen to tackle 1Work is done during internship at Microsoft STCA. 2Corresponding authors. Method en es zh tr zero-shot 88.24 52.18 30.01 3.08 code-switching 88.69 54.42 45.24 7.41 Table 1: mBERT based zero-shot and code-switching (CoSDA-ML) results on four languages of MultiATIS++ (semantic EM accuracy). this problem that transfer the language-agnostic knowledge from high-resource (source) languages to low-resource (target) languages. For the data-based transfer methods, machine translation is first applied to translate the source ut- terances into the targets (Upadhyay et al., 2018; Schuster et al., 2019; Xu et al., 2020). How- ever, machine translation may be unreliable or unavailable for some extremely low-resource lan- guages (Upadhyay et al., 2018). Therefore, multi- lingual code-switching (Liu et al., 2020a; Qin et al., 2020) is developed to reduce the dependency on machine translation, which simply uses bilingual dictionaries to randomly select some words in the utterance to be replaced by the translation words in other languages. Code-switching has achieved promising results as the word representations in the …

Content truncated to 500 words