Modeling User Satisfaction Dynamics in Dialogue via Hawkes Process

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8875–8889 July 9-14, 2023 ©2023 Association for Computational Linguistics Modeling User Satisfaction Dynamics in Dialogue via Hawkes Process Fanghua Ye† and Zhiyuan Hu‡ and Emine Yilmaz† †University College London ‡National University of Singapore {fanghua.ye.19, emine.yilmaz}@ucl.ac.uk, zhiyuan_hu@u.nus.edu Abstract Dialogue systems have received increasing at- tention while automatically evaluating their per- formance remains challenging. User satisfac- tion estimation (USE) has been proposed as an alternative. It assumes that the performance of a dialogue system can be measured by user satisfaction and uses an estimator to simulate users. The effectiveness of USE depends heav- ily on the estimator. Existing estimators inde- pendently predict user satisfaction at each turn and ignore satisfaction dynamics across turns within a dialogue. In order to fully simulate users, it is crucial to take satisfaction dynamics into account. To fill this gap, we propose a new estimator ASAP (sAtisfaction eStimation via HAwkes Process) that treats user satisfaction across turns as an event sequence and employs a Hawkes process to effectively model the dy- namics in this sequence. Experimental results on four benchmark dialogue datasets demon- strate that ASAP can substantially outperform state-of-the-art baseline estimators. 1 Introduction Dialogue systems are playing an increasingly im- portant role in our daily lives. They can serve as intelligent assistants to help users accomplish tasks and answer questions or as social companion bots to converse with users for entertainment (Ni et al., 2022; Fu et al., 2022). In recent years, the research and development of dialogue systems has made remarkable progress. However, due to the complex- ity of human communication, the latest dialogue systems may still fail to understand users’ intents and generate inappropriate responses (Liang et al., 2021; Deng and Lin, 2022; Pan et al., 2022). These deficiencies pose huge challenges to deploying dia- logue systems to real-life applications, especially high-stakes ones such as finance and health. In light of this, it is crucial to evaluate the performance of dialogue systems adequately in their development phase (Sun et al., 2021; Deriu et al., 2021). Hello, how may I help you today? I want to cancel my handbag order. It can only be canceled if the following conditions are met… Yes, I meet all the conditions. Good. You have two orders. Which one do you want to cancel? Wait, didn't I just mention that I want to cancel the handbag order? Figure 1: An example dialogue showing the dynamics of user satisfaction across different interaction turns. Generally speaking, there are two types of eval- uation methods, human evaluation and automatic evaluation (Deriu et al., 2021). Human evaluation is fairly effective, but costly and hard to scale up. By contrast, automatic evaluation is more scalable. However, due to the ambiguity of what constitutes a high-quality dialogue, there are currently no uni- versally accepted evaluation metrics. Existing com- monly used metrics such as BLEU (Papineni et al., 2002) usually do not agree with human judgment. Nonetheless, user satisfaction estimation …

Content truncated to 500 words