Show, Don’t Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4541 - 4549 July 10-15, 2022 ©2022 Association for Computational Linguistics Show, Don’t Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue Raghav Gupta∗, Harrison Lee∗, Jeffrey Zhao, Abhinav Rastogi, Yuan Cao, Yonghui Wu Google Research {raghavgupta, harrisonlee}@google.com Abstract Building universal dialogue systems that op- erate across multiple domains/APIs and gen- eralize to new ones with minimal overhead is a critical challenge. Recent works have lever- aged natural language descriptions of schema elements to enable such systems; however, de- scriptions only indirectly convey schema se- mantics. In this work, we propose Show, Don’t Tell, which prompts seq2seq models with a la- beled example dialogue to show the semantics of schema elements rather than tell the model through descriptions. While requiring simi- lar effort from service developers as generat- ing descriptions, we show that using short ex- amples as schema representations with large language models results in state-of-the-art per- formance on two popular dialogue state track- ing benchmarks designed to measure zero- shot generalization - the Schema-Guided Di- alogue dataset and the MultiWOZ leave-one- out benchmark. 1 Introduction Task-oriented dialogue (TOD) systems need to sup- port an ever-increasing variety of services. Since many service developers lack the resources to col- lect data and train models, zero and few-shot trans- fer to unseen services is critical to the democratiza- tion of dialogue agents. Recent approaches to generalizable TOD sys- tems primarily rely on combining two techniques: large language models like BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020), and schema- guided modeling - i.e. using natural language de- scriptions of schema elements (intents and slots) as model inputs to enable transfer to unseen services (Rastogi et al., 2020a,b). Models combining the two currently hold state-of-the-art (SotA) results on dialogue state tracking (DST) (Heck et al., 2020; Lee et al., 2021a; Zhao et al., 2022). ∗*Equal contribution However, description-based schema representa- tions have some drawbacks. Writing precise natural language descriptions requires manual effort and can be difficult to write succinctly. Also, descrip- tions only provide indirect supervision about how to interact with a service compared to an exam- ple. Furthermore, Lee et al. (2021b) showed that schema-guided DST models are not robust to vari- ations in schema descriptions, causing significant quality drops. We propose using a single dialogue example with state annotations as an alternative to the description-based schema representation, similar to one-shot priming (Brown et al., 2020) - an ap- proach we call Show, Don’t Tell (SDT). Through demonstration, we show models the schema seman- tics rather than tell them through natural language descriptions, as seen in Figure 1. SDT achieves SotA accuracy and generalization to new APIs across both the Schema-Guided Dataset (SGD) (Rastogi et al., 2020b) and MultiWOZ Leave-One- Out (Budzianowski et al., 2018; Lin et al., 2021b) benchmarks, while being more data-efficient and robust to schema variations. 2 Show, Don’t Tell Following SoTA models, we pose DST …

Content truncated to 500 words