Improving Limited Labeled Dialogue State Tracking with Self-Supervision

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4462–4472 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4462 Improving Limited Labeled Dialogue State Tracking with Self-Supervision Chien-Sheng Wu, Steven Hoi, and Caiming Xiong Salesforce Research [wu.jason, shoi, cxiong]@salesforce.com Abstract Existing dialogue state tracking (DST) models require plenty of labeled data. However, col- lecting high-quality labels is costly, especially when the number of domains increases. In this paper, we address a practical DST problem that is rarely discussed, i.e., learning efficiently with limited labeled data. We present and in- vestigate two self-supervised objectives: pre- serving latent consistency and modeling con- versational behavior. We encourage a DST model to have consistent latent distributions given a perturbed input, making it more robust to an unseen scenario. We also add an auxil- iary utterance generation task, modeling a po- tential correlation between conversational be- havior and dialogue states. The experimental results show that our proposed self-supervised signals can improve joint goal accuracy by 8.95% when only 1% labeled data is used on the MultiWOZ dataset. We can achieve an additional 1.76% improvement if some unla- beled data is jointly trained as semi-supervised learning. We analyze and visualize how our proposed self-supervised signals help the DST task and hope to stimulate future data-efficient DST research. 1 Introduction Dialogue state tracking is an essential compo- nent in task-oriented dialogue systems designed to extract user goals/intentions expressed dur- ing a conversation. Accurate DST performance can facilitate downstream applications such as dialogue management. However, collecting di- alogue state labels is very expensive and time- consuming (Budzianowski et al., 2018), requiring dialogue experts or trained turkers to indicate all (domain, slot, value) information for each turn in dialogues. This problem becomes important from single-domain to multi-domain scenarios. It will be more severe for a massive-multi-domain setting, making DST models less scalable to a new domain. Existing DST models require plenty of state labels, especially those ontology-based DST ap- proaches (Henderson et al., 2014; Mrkˇsi´c et al., 2017; Zhong et al., 2018). They assume a pre- defined ontology that lists all possible values is available, but an ontology requires complete state annotation and is hard to get in real scenario (Xu and Hu, 2018). They also cannot track unseen slot values that are not predefined. Ontology-free ap- proaches (Xu and Hu, 2018; Chao and Lane, 2019), on the other hand, are proposed to generate slot values from dialogue history directly. They achieve good performance on multi-domain DST by copy- attention mechanism but still observe a significant performance drop under limited labeled data sce- nario (Wu et al., 2019a). In this paper, we approach the DST problem us- ing copy-augmented ontology-free models from a rarely discussed perspective, assuming that only a few dialogues in a dataset have annotated state labels. We present two self-supervised learning (SSL) solutions: 1) Preserving latent consistency: We encourage a DST model to have similar la- tent distributions (e.g., attention weights and hid- den states) for a set of slightly perturbed inputs. This …

Content truncated to 500 words