Incremental Learning from Scratch for Task-Oriented Dialogue Systems

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3710–3720 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3710 Incremental Learning from Scratch for Task-Oriented Dialogue Systems Weikang Wang1,2, Jiajun Zhang1,2, Qian Li3, Mei-Yuh Hwang3, Chengqing Zong1,2,4 and Zhifei Li3 1 National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China 2 University of Chinese Academy of Sciences, Beijing, China 3 Mobvoi, Beijing, China 4 CAS Center for Excellence in Brain Science and Intelligence Technology, Beijing, China {weikang.wang, jjzhang, cqzong}@nlpr.ia.ac.cn {qli, mhwang, zfli}@mobvoi.com Abstract Clarifying user needs is essential for exist- ing task-oriented dialogue systems. How- ever, in real-world applications, developers can never guarantee that all possible user de- mands are taken into account in the design phase. Consequently, existing systems will break down when encountering unconsidered user needs. To address this problem, we pro- pose a novel incremental learning framework to design task-oriented dialogue systems, or for short Incremental Dialogue System (IDS), without pre-defining the exhaustive list of user needs. Specifically, we introduce an uncer- tainty estimation module to evaluate the con- fidence of giving correct responses. If there is high confidence, IDS will provide responses to users. Otherwise, humans will be involved in the dialogue process, and IDS can learn from human intervention through an online learning module. To evaluate our method, we propose a new dataset which simulates unanticipated user needs in the deployment stage. Experi- ments show that IDS is robust to unconsidered user actions, and can update itself online by smartly selecting only the most effective train- ing data, and hence attains better performance with less annotation cost.1 1 Introduction Data-driven task-oriented dialogue systems have been a focal point in both academic and industry research recently. Generally, the first step of build- ing a dialogue system is to clarify what users are allowed to do. Then developers can collect data to train dialogue models to support the defined ca- pabilities. Such systems work well if all possi- ble combinations of user inputs and conditions are considered in the training stage (Paek and Pierac- cini, 2008; Wang et al., 2018). However, as shown 1https://github.com/Leechikara/ Incremental-Dialogue-System What should I do to update the operating system? Our products support Android and iOS. Which one do you prefer? Hi, I can help you find the most suitable product. Figure 1: An example of task-oriented dialogue sys- tem. The system is designed to guide users to find a suitable product. Thus, when encountering unconsid- ered user needs such as ”how to update the operating system”, the system will give unreasonable responses. in Fig. 1, if users have unanticipated needs, the system will give unreasonable responses. This phenomenon is mainly caused by a biased understanding of real users. In fact, before system deployment, we do not know what the customers will request of the system. In general, this prob- lem can be alleviated by more detailed user stud- ies. But we can never guarantee that all user needs are considered in …

Content truncated to 500 words