A Few-Shot Semantic Parser for Wizard-of-Oz Dialogues with the Precise ThingTalk Representation

Findings of the Association for Computational Linguistics: ACL 2022, pages 4021 - 4034 May 22-27, 2022 c⃝2022 Association for Computational Linguistics A Few-Shot Semantic Parser for Wizard-of-Oz Dialogues with the Precise ThingTalk Representation Giovanni Campagna Sina J. Semnani Ryan Kearns Lucas Jun Koba Sato Silei Xu Monica S. Lam Computer Science Department Stanford University Stanford, CA, USA {gcampagn,sinaj,kearns,satojk,silei,lam}@cs.stanford.edu Abstract Previous attempts to build effective seman- tic parsers for Wizard-of-Oz (WOZ) conversa- tions suffer from the difficulty in acquiring a high-quality, manually annotated training set. Approaches based only on dialogue synthesis are insufficient, as dialogues generated from state-machine based models are poor approx- imations of real-life conversations. Further- more, previously proposed dialogue state rep- resentations are ambiguous and lack the preci- sion necessary for building an effective agent. This paper proposes a new dialogue represen- tation and a sample-efficient methodology that can predict precise dialogue states in WOZ conversations. We extended the ThingTalk rep- resentation to capture all information an agent needs to respond properly. Our training strat- egy is sample-efficient: we combine (1) few- shot data sparsely sampling the full dialogue space and (2) synthesized data covering a sub- set space of dialogues generated by a succinct state-based dialogue model. The completeness of the extended ThingTalk language is demon- strated with a fully operational agent, which is also used in training data synthesis. We demonstrate the effectiveness of our methodology on MultiWOZ 3.0, a reannota- tion of the MultiWOZ 2.1 dataset in ThingTalk. ThingTalk can represent 98% of the test turns, while the simulator can emulate 85% of the validation set. We train a contextual seman- tic parser using our strategy, and obtain 79% turn-by-turn exact match accuracy on the rean- notated test set.1 1 Introduction Virtual assistants and task-oriented dialogue agents are transforming how consumers interact with com- puters. This has led to active research on dialogue state tracking networks (Ren et al., 2019; Zhou and 1Our data and code can be downloaded from https:// oval.cs.stanford.edu/releases/ SearchQuestion: area; Restaurant, food = “indian” && price == cheap { name = “Kohinoor”, area = centre, ... } { name = “Royal Spice”, area = north, ... } “Do you have anything with Indian food?” user utterance RecommendMany; Restaurant, price == cheap { name = “Pizza Hut City Centre”, area = centre, ... } { name = “The Missing Sock”, area = east, ... } input formal dialogue CSP Model Exec: Restaurant, food = “indian” && price == cheap user state Agent “Do you have a specific part of town in mind?” agent utterance output formal dialogue ... ... Figure 1: The inference-time flow of a dialogue agent with a contextual semantic parser based on the ThingTalk representation. Small, 2019; Zhang et al., 2020; Chen et al., 2020; Heck et al., 2020), and even full neural networks that track dialogue states, implement dialogue poli- cies, and generate agent utterances (Williams and Zweig, 2016; Eric et al., 2017; Zhang et al., 2020; Peng et al., 2020; Hosseini-Asl et al., 2020). Dialogue state tracking …

Content truncated to 500 words