A Comparative Study on Schema-Guided Dialogue State Tracking

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 782–796 June 6–11, 2021. ©2021 Association for Computational Linguistics 782 A Comparative Study on Schema-Guided Dialogue State Tracking Jie Cao ∗ School of Computing, University of Utah jcao@cs.utah.edu Yi Zhang AWS AI, Amazon yizhngn@amazon.com Abstract Frame-based state representation is widely used in modern task-oriented dialog systems to model user intentions and slot values. How- ever, a fixed design of domain ontology makes it difficult to extend to new services and APIs. Recent work proposed to use natural language descriptions to define the domain ontology in- stead of tag names for each intent or slot, thus offering a dynamic set of schema. In this paper, we conduct in-depth comparative stud- ies to understand the use of natural language description for schema in dialog state track- ing. Our discussion mainly covers three as- pects: encoder architectures, impact of sup- plementary training, and effective schema de- scription styles. We introduce a set of newly designed bench-marking descriptions and re- veal the model robustness on both homoge- neous and heterogeneous description styles in training and evaluation. 1 Introduction From early frame-driven dialog system GUS (Bo- brow et al., 1977) to virtual assistants (Alexa, Siri, and Google Assistant et al.), frame-based dialog state tracking has long been studied to meet various challenges. In particular, how to support an ever- increasing number of services and APIs spanning multiple domains has been a focal point in recent years, evidenced by multi-domain dialog model- ing (Budzianowski et al., 2018; Byrne et al., 2019; Shah et al., 2018a) and transferable dialog state tracking to unseen intent/slots (Mrkši´c et al., 2017; Wu et al., 2019; Hosseini-Asl et al., 2020). Recently, Rastogi et al. (2019) proposed a new paradigm called schema-guided dialog for trans- ferable dialog state tracking by using natural lan- guage description to define a dynamic set of service schemata. As shown in Figure 1, the primary moti- vation is that these descriptions can offer effective ∗Work done when Jie Cao was an intern at Amazon Figure 1: An example dialog from Restaurant_1 ser- vice, along with its service/intent/slot descriptions and dialog state representation. knowledge sharing across different services, e.g., connecting semantically similar concepts across heterogeneous APIs, thus allowing a unified model to handle unseen services and APIs. With the pub- licly available schema-guided dialog dataset (SG- DST henceforward) as a testbed, they organized a state tracking shared task composed of four sub- tasks: intent classfication (Intent), requested slot identification (Req), categorical slot labeling (Cat), and noncategorical slot labeling (NonCat) (Rastogi et al., 2020). Many participants achieved promising performance by exploiting the schema description for dialog modeling, especially on unseen services. Despite the novel approach and promising re- sults, current schema-guided dialog state tracking 783 task only evaluates on a single dataset with lim- ited variation in schema definition. It is unknown how this paradigm generalizes to other datasets and other different styles of descriptions. In this paper, we …

Content truncated to 500 words