Out-of-Task Training for Dialog State Tracking Models

Proceedings of the 28th International Conference on Computational Linguistics, pages 6767–6774 Barcelona, Spain (Online), December 8-13, 2020 6767 Out-of-Task Training for Dialog State Tracking Models Michael Heck, Christian Geishauser, Hsien-Chin Lin, Nurul Lubis, Marco Moresi, Carel van Niekerk, Milica Gaˇsi´c Heinrich Heine University D¨usseldorf, Germany {heckmi,geishaus,linh,lubis,moresi,niekerk,gasic}@hhu.de Abstract Dialog state tracking (DST) suffers from severe data sparsity. While many natural language processing (NLP) tasks benefit from transfer learning and multi-task learning, in dialog these methods are limited by the amount of available data and by the specificity of dialog applications. In this work, we successfully utilize non-dialog data from unrelated NLP tasks to train dialog state trackers. This opens the door to the abundance of unrelated NLP corpora to mitigate the data sparsity issue inherent to DST. 1 Introduction The role of the dialog state tracker in a task-oriented dialog system is to summarise the history of the conversation so far and extract the user goal. Dialog state tracking (DST) suffers extraordinarily from data sparsity. Collecting data for DST is expensive and time consuming. Typically, conversations are either staged or collected in a Wizard-of-Oz style setup and annotated by hand, severely inhibiting data collection. The enormous number of possible dialog states further exacerbates this. Even if we combined all data that commercial assistants generate, there will still be realistic but unobserved dialog states. Unsupervised learning, transfer learning and multi-task learning (MTL) (Caruana, 1997) in general help mitigate data sparsity. Unsupervised learning relies on predicting inherent characteristics of the data. Transfer learning exploits knowledge learned on related problems to generalize to new tasks. MTL optimizes towards solving multiple tasks at once for synergy effects. Typically, utilized datasets have related domains and tasks share objectives. In other words, these strategies are typically used to address the problem of adaptation. Recent approaches to adaptation in NLP tasks rely on contextual models. The methods above have been applied to improve generalization across related tasks and datasets. For instance, Phang et al. (2018), Wang et al. (2019) and Pruksachatkun et al. (2020) facilitate transfer learning by intermediate task fine-tuning (ITFT) on tasks that are related to the target task. Peng et al. (2020) and Liu et al. (2019a) jointly optimize transformer based models (Vaswani et al., 2017) towards multiple related tasks and/or domains. The latter apply MTL to pre-training, rather than fine-tuning. Gururangan et al. (2020) report improvements by continuing unsupervised pre-training for domain/task adaptation. Raffel et al. (2019) and Keskar et al. (2019) propose model architectures that handle diverse tasks with a unified mechanism. Natural language understanding (NLU) and dialog state tracking (DST) benefit from joint model- ing via multitask learning, as this utilizes dialog data more efficiently (Rastogi et al., 2018). Recent approaches view DST as a generative problem (Wu et al., 2019; Ren et al., 2019) or as a reading compre- hension problem (Gao et al., 2019; Chao and Lane, 2019) utilizing contextual models. In the latter, span prediction or sequence tagging extract relevant information from the input directly. These mechanisms utilize training …

Content truncated to 500 words