Cross-lingual Feature Extraction from Monolingual Corpora for Low-resource Unsupervised Bilingual Lexicon Induction

Proceedings of the 29th International Conference on Computational Linguistics, pages 553–569 October 12–17, 2022. 553 SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding Wanwei He1,2, , Yinpei Dai3, , Binyuan Hui3, , Min Yang1∗, Zheng Cao3, Jianbo Dong3, Fei Huang3, Luo Si3, Yongbin Li3∗ 1Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China 2University of Chinese Academy of Sciences, China 3DAMO Academy, Alibaba Group {ww.he, min.yang}@siat.ac.cn {yinpei.dyp, f.huang, luo.si, shuide.lyb}@alibaba-inc.com Abstract Pre-training methods with contrastive learning objectives have shown remarkable success in dialog understanding tasks. However, current contrastive learning solely considers the self-augmented dialog samples as positive samples and treats all other dialog samples as negative ones, which enforces dissimilar representations even for dialogs that are semantically related. In this paper, we propose SPACE-2, a tree-structured pre-trained conversation model, which learns dialog representations from limited labeled dialogs and large-scale unlabeled dialog corpora via semi-supervised contrastive pre-training. Concretely, we first define a general semantic tree structure (STS) to unify the inconsistent annotation schema across different dialog datasets, so that the rich structural information stored in all labeled data can be exploited. Then we propose a novel multi-view score function to increase the relevance of all possi- ble dialogs that share similar STSs and only push away other completely different dialogs during supervised contrastive pre-training. To fully exploit unlabeled dialogs, a basic self-supervised contrastive loss is also added to refine the learned representations. Experiments show that our method can achieve new state-of- the-art results on the DialoGLUE benchmark consisting of seven datasets and four popular dialog understanding tasks. For reproducibility, we release the code and data at https: //github.com/AlibabaResearch/ DAMO-ConvAI/tree/main/space-2. 1 Introduction Task-oriented dialog (TOD) systems aim to help users to accomplish specific tasks through natural language interactions, e.g., restaurant booking, ho- tel reserving, and movie searching (Young et al., 2013). In order to fulfill a user goal, the dialog ∗ Corresponding authors. Equal contribution. systems must be capable of extracting structured semantics from the dialog utterances, which is of- ten referred to as dialog understanding. Common tasks of dialog understanding include: 1) intent prediction, where the system classifies the user in- tention given the current utterance, 2) slot filling, where the system extracts the value spans for pre- defined slots from user utterances, and 3) dialog state tracking, where slot-value pairs are predicted and updated according to multi-turn dialog history. Recent methods address the dialog understand- ing problems mainly by task-adaptive training with pre-trained language models (PLMs) (Henderson et al., 2020; Mehri et al., 2020; Zhang et al., 2022). By fine-tuning the entire parameters of PLMs on dialog corpora, the versatile knowledge acquired from large-scale corpora can be adapted to han- dle various dialog understanding tasks. For ex- ample, TOD-BERT (Wu et al., 2020) continually pre-trained a BERT on large-scale task-oriented di- alog corpora with similar objectives to grasp more knowledge in conversations and achieved good re- sults on a wide range of tasks. To further enhance the representations of dialogs, contrastive learning is widely used for dialog pre-training …

Content truncated to 500 words