MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3391–3405, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3391 MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Pascale Fung Center for Artificial Intelligence Research (CAiRE) Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong zlinao@connect.ust.hk Abstract In this paper, we propose Minimalist Trans- fer Learning (MinTL) to simplify the system design process of task-oriented dialogue sys- tems and alleviate the over-dependency on an- notated data. MinTL is a simple yet effective transfer learning framework, which allows us to plug-and-play pre-trained seq2seq models, and jointly learn dialogue state tracking and di- alogue response generation. Unlike previous approaches, which use a copy mechanism to “carryover” the old dialogue states to the new one, we introduce Levenshtein belief spans (Lev), that allows efficient dialogue state track- ing with a minimal generation length. We in- stantiate our learning framework with two pre- trained backbones: T5 (Raffel et al., 2019) and BART (Lewis et al., 2019), and evaluate them on MultiWOZ. Extensive experiments demonstrate that: 1) our systems establish new state-of-the-art results on end-to-end response generation, 2) MinTL-based systems are more robust than baseline methods in the low re- source setting, and they achieve competitive re- sults with only 20% training data, and 3) Lev greatly improves the inference efficiency1. 1 Introduction Building robust task-oriented dialogue systems is challenging due to complex system design and limited availability of human-annotated data (Wen et al., 2017; Wu et al., 2019b). A dialogue agent is expected to learn dialogue reasoning, decision making, and language generation, which require a large amount of training data. However, col- lecting and annotating data for training a dia- logue system is time-intensive and not transferable among domains (Young et al., 2013). One possible workaround is to leverage the pre-trained language 1Code available in https://github.com/zlinao/ MinTL model to reduce human supervision (Budzianowski and Vuli´c, 2019). Recent progress in pre-training language mod- els has been shown to be promising in alleviat- ing the data scarcity problem (Budzianowski and Vuli´c, 2019; Wu et al., 2020). Such models are typically pre-trained on large-scale plain text with self-supervised objectives, e.g., language model- ing (Radford et al., 2019) and language denois- ing (Devlin et al., 2019). Fine tuning pre-trained language models improves a wide range of natu- ral language processing applications (Lewis et al., 2019; Raffel et al., 2019), notably machine trans- lation (Conneau and Lample, 2019), and person- alized dialogue response generation (Wolf et al., 2019b). However, adapting pre-trained language models to task-oriented dialogue systems is not triv- ial. Current state-of-the-art (SOTA) approaches in task-oriented dialogue rely on several tasks-specific modules, such as State Operation Predictor (Kim et al., 2019) for dialogue state tracking, and Copy- Net (Gu et al., 2016) for end-to-end dialogue task completion (Lei et al., 2018; Zhang et al., 2019b). Such modules are usually absent in the pre-training stage. Therefore, tasks-specific architecture modi- …

Content truncated to 500 words