GL-CLeF: A Global–Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2677 - 2686 May 22-27, 2022 c⃝2022 Association for Computational Linguistics GL-CLEF: A Global–Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding Libo Qin1∗, Qiguang Chen1, Tianbao Xie1, Qixin Li1, Jian-Guang Lou2 , Wanxiang Che1†, Min-Yen Kan3 1Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China 2 Microsoft Research Asia, Beijing, China 3Department of Computer Science, National University of Singapore {lbqin,tianbaoxie,qxli,car}@ir.hit.edu.cn; jlou@microsoft.com; kanmy@comp.nus.edu.sg Abstract Due to high data demands of current meth- ods, attention to zero-shot cross-lingual spo- ken language understanding (SLU) has grown, as such approaches greatly reduce human an- notation effort. However, existing models solely rely on shared parameters, which can only perform implicit alignment across lan- guages. We present Global–Local Contrastive LEarning Framework (GL-CLEF) to address this shortcoming. Specifically, we employ con- trastive learning, leveraging bilingual dictio- naries to construct multilingual views of the same utterance, then encourage their represen- tations to be more similar than negative exam- ple pairs, which achieves to explicitly aligned representations of similar sentences across lan- guages. In addition, a key step in GL-CLEF is a proposed Local and Global compo- nent, which achieves a fine-grained cross- lingual transfer (i.e., sentence-level Local in- tent transfer, token-level Local slot transfer, and semantic-level Global transfer across in- tent and slot). Experiments on MultiATIS++ show that GL-CLEF achieves the best perfor- mance and successfully pulls representations of similar sentences across languages closer. 1 Introduction Spoken language understanding (SLU) is a critical component in task-oriented dialogue systems (Tur and De Mori, 2011; Qin et al., 2021b). It usu- ally includes two sub-tasks: intent detection to identify users’ intents and slot filling to extract semantic constituents from the user’s query. With the advent of deep neural network methods, SLU has met with remarkable success. However, ex- isting SLU models rely on large amounts of an- notated data, which makes it hard to scale to low- resource languages that lack large amounts of la- beled data. To address this shortcoming, zero-shot ∗Work done during internship at Microsoft Research Asia and remote visiting at National University of Singapore. †Email corresponding. GL-CLEF watch action movie 看 动作 电影 movie 电影 [CLS] 电影 Sentence-level Local Intent Alignment Token-level Local Slot Alignment Semantic-level Global Intent-Slot Alignment (b) GL-CLEF Shared Parameters (a) Prior work. movie 电影 Explicit Alignment Implicit Alignment closer! closer! closer! [CLS] [CLS] [CLS] [CLS] [CLS] [CLS] watch action movie 看 动作 电影 [CLS] [CLS] Figure 1: (a) Prior work (Implicit Alignment); (b) GL- CLEF (Explicit Alignment). Different color denotes representations across different languages. [CLS] rep- resents the sentence representation. cross-lingual SLU generalization leverages the la- beled training data in high-resource languages to transfer the trained model to a target, low-resource language, which gains increasing attention. To this end, many works have been explored for zero-shot cross-lingual SLU. Multilingual BERT (mBERT) (Devlin et al., 2019), a cross-lingual contextual pre-trained model from a large amount of multi-lingual corpus multi-lingual corpus, has …

Content truncated to 500 words