More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Ranking

Findings of the Association for Computational Linguistics: EACL 2023, pages 1443–1454 May 2-6, 2023 ©2023 Association for Computational Linguistics More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Ranking Alexandru Coca†, Bo-Hsiang Tseng‡, Weizhe Lin†, Bill Byrne† †Department of Engineering, University of Cambridge, United Kingdom ‡Apple {ac2123, wl356, wjb31}@cam.ac.uk bohsiang_tseng@apple.com Abstract The schema-guided paradigm overcomes scala- bility issues inherent in building task-oriented dialogue (TOD) agents with static ontologies. Instead of operating on dialogue context alone, agents have access to hierarchical schemas con- taining task-relevant natural language descrip- tions. Fine-tuned language models excel at schema-guided dialogue state tracking (DST) but are sensitive to the writing style of the schemas. We explore methods for improving the robustness of DST models. We propose a framework1 for generating synthetic schemas which uses tree-based ranking to jointly opti- mise lexical diversity and semantic faithfulness. The generalisation of strong baselines is im- proved when augmenting their training data with prompts generated by our framework, as demonstrated by marked improvements in av- erage joint goal accuracy (JGA) and schema sensitivity (SS) on the SGD-X benchmark. 1 Introduction DST is concerned with tracking user goals in task- oriented conversations. The goals are represented as key-value pair sequences, with the keys known as slots (e.g. hotel name). Pre-trained language models (PLMs) (Devlin et al., 2019; Raffel et al., 2020) have helped shift focus from systems that can only track slots drawn from a database or domain ontology (Henderson et al., 2014) to models that do not require re-training to parse goals in new domains. The Schema-Guided Dialogue (SGD) dataset (Rastogi et al., 2020) facilitates this shift with a large-scale set of conversations grounded in 45 service APIs or schemas that describe the domains, slots and user intents that annotate the conversations (Appendix A). Test set dialogues are grounded in 6 schemas seen during training and 15 unseen ones. Neural models perform impressively on the diffi- cult schema-guided DST task (Rastogi et al., 2020), 1Code will be released here: https://bit.ly/3WYB7Fl but Lee et al. (2022) show that the uniformity of the descriptive language of the schemas facilitates this. They create the SGD-X benchmark to evalu- ate robust zero-shot generalisation of DST models. This is achieved by grounding the SGD test set conversations in five schema variants increasingly dissimilar to the SGD schemata2. To perform well, a DST model should correctly track the state of a dialogue when conditioned, in turn, on prompts constructed from the five variants. We show how to improve DST robustness by introducing controlled variability in the data. We contribute to robust DST research by (1) a flexi- ble framework for generating and ranking diverse outputs of a paraphrase model based on a tree- clustering algorithm designed to control lexical di- versity and semantic similarity; (2) combine state- of-the-art paraphrase models and language gen- eration metrics to generate increasingly diverse schemata paraphrases; (3) show that augmenting the training dataset with these schemata improves the robustness and generalisation performance of strong DST baselines. 2 Related Work Input variety, …

Content truncated to 500 words