Novel Slot Detection: A Benchmark for Discovering Unknown Slot Types in the Task-Oriented Dialogue System

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3484–3494 August 1–6, 2021. ©2021 Association for Computational Linguistics 3484 Novel Slot Detection: A Benchmark for Discovering Unknown Slot Types in the Task-Oriented Dialogue System Yanan Wu1∗, Zhiyuan Zeng1∗, Keqing He2∗, Hong Xu1 Yuanmeng Yan1, Huixing Jiang2, Weiran Xu1∗ 1Pattern Recognition & Intelligent System Laboratory 1Beijing University of Posts and Telecommunications, Beijing, China 2Meituan Group, Beijing, China {yanan.wu,zengzhiyuan,xuhong,yanyuanmeng,xuweiran}@bupt.edu.cn {hekeqing,jianghuixing}@meituan.com Abstract Existing slot filling models can only recognize pre-defined in-domain slot types from a lim- ited slot set. In the practical application, a reliable dialogue system should know what it does not know. In this paper, we introduce a new task, Novel Slot Detection (NSD), in the task-oriented dialogue system. NSD aims to discover unknown or out-of-domain slot types to strengthen the capability of a dialogue sys- tem based on in-domain training data. Be- sides, we construct two public NSD datasets, propose several strong NSD baselines, and es- tablish a benchmark for future work. Finally, we conduct exhaustive experiments and quali- tative analysis to comprehend key challenges and provide new guidance for future direc- tions1. 1 Introduction Slot filling plays a vital role to understand user queries in personal assistants such as Amazon Alexa, Apple Siri, Google Assistant, etc. It aims at identifying a sequence of tokens and extracting semantic constituents from the user queries. Given a large scale pre-collected training corpus, existing neural-based models (Mesnil et al., 2015; Liu and Lane, 2015, 2016; Goo et al., 2018; Haihong et al., 2019; Chen et al., 2019; He et al., 2020b,d; Yan et al., 2020; Louvan and Magnini, 2020; He et al., 2020a) have been actively applied to slot filling and achieved promising results. Existing slot filling models can only recognize pre-defined entity types from a limited slot set, which is insufficient in the practical application scenario. A reliable slot filling model should not only predict the pre-defined slots but also detect po- tential unknown slot types to know what it doesn’t ∗The first three authors contribute equally. Weiran Xu is the corresponding author. 1https://github.com/ChestnutWYN/ACL20 21-Novel-Slot-Detection Agent: What can I do for you? User: Play is this my world by leo arnaud. Play is this my world by leo arnaud. playlist artist Dialogue System DM & NLG Module is this my world is an unknown slot type (denoted as NS). It’s the name of leo arnaud’s album. Dialogue System without Novel Slot Detector Agent: What can I do for you? User: Play is this my world by leo arnaud. Dialogue System DM & NLG Module Dialogue System with Novel Slot Detector NLU Module Play is this my world by leo arnaud. NS artist Novel Slot Detector Human Annotated Collected Novel Slots Update Dialogue System Agent: “is this my world” is probably a novel slot. The current system can not handle it. Agent: You don’t have a playlist called “is this my world” × √ Figure 1: An example of …

Content truncated to 500 words