Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2079–2091 July 9-14, 2023 ©2023 Association for Computational Linguistics Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues Yue Feng †∗ Yunlong Jiao ‡ Animesh Prasad ‡ Nikolaos Aletras ◇‡ Emine Yilmaz †‡ Gabriella Kazai ‡ †University College London, London, UK ‡Amazon, London, United Kingdom ◇University of Sheffield, Sheffield, UK †{yue.feng.20,emine.yilmaz}@ucl.ac.uk ‡{jyunlong,gkazai}@amazon.co.uk ◇n.aletras@sheffield.ac.uk Abstract User Satisfaction Modeling (USM) is one of the popular choices for task-oriented dialogue systems evaluation, where user satisfaction typ- ically depends on whether the user’s task goals were fulfilled by the system. Task-oriented di- alogue systems use task schema, which is a set of task attributes, to encode the user’s task goals. Existing studies on USM neglect explic- itly modeling the user’s task goals fulfillment using the task schema. In this paper, we pro- pose SG-USM, a novel schema-guided user satisfaction modeling framework. It explicitly models the degree to which the user’s prefer- ences regarding the task attributes are fulfilled by the system for predicting the user’s satis- faction level. SG-USM employs a pre-trained language model for encoding dialogue context and task attributes. Further, it employs a ful- fillment representation layer for learning how many task attributes have been fulfilled in the dialogue, an importance predictor component for calculating the importance of task attributes. Finally, it predicts the user satisfaction based on task attribute fulfillment and task attribute importance. Experimental results on bench- mark datasets (i.e. MWOZ, SGD, ReDial, and JDDC) show that SG-USM consistently out- performs competitive existing methods. Our extensive analysis demonstrates that SG-USM can improve the interpretability of user satis- faction modeling, has good scalability as it can effectively deal with unseen tasks and can also effectively work in low-resource settings by leveraging unlabeled data.1 1 Introduction Task-oriented dialogue systems have emerged for helping users to solve specific tasks effi- ciently (Hosseini-Asl et al., 2020). Evaluation is ∗Work done while Yue Feng was an intern at Amazon, Alexa Shopping. 1Code is available at https://github.com/amzn/ user-satisfaction-modeling. Dialogue User’s Task Goal Schema for Restaurant Task I’d like to look for a diner in Vacaville. I am searching for one that is intermediate priced. Japanese Restaurant is a lovely diner around there. That's prefect! Thanks! It's my pleasure. Task Attributes: “City”: City in which the restaurant is located. “Price_Range”: Price range for the restaurant. “City”: Vacaville “Price_Range”: Intermediate Solution User System User System User Satisfaction Level Satisfied §Japanese Restaurant: “City”: Vacaville § “Price_Range”: Intermediate Figure 1: Task-oriented dialogue system has a prede- fined schema for each task, which is composed of a set of task attributes. In a dialogue, the user’s task goal is encoded by the task attribute and value pairs. The user is satisfied with the service when the provided solution fulfills the user’s preferences for the task attributes. a crucial part of the development process of such systems. Many of the standard automatic evalu- ation metrics, e.g. BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), have been shown …

Content truncated to 500 words