Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3897–3909 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3897 Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations Praveen Kumar Bodigutla pbodigutla@linkedin.com∗ Aditya Tiwari aditiwar@amazon.com Josep Valls Vargas jvalls@amazon.com Lazaros Polymenakos polyml@amazon.com Spyros Matsoukas matsouka@amazon.com Abstract Dialogue level quality estimation is vital for optimizing data driven dialogue management. Current automated methods to estimate turn and dialogue level user satisfaction employ hand-crafted features and rely on complex an- notation schemes, which reduce the general- izability of the trained models. We propose a novel user satisfaction estimation approach which minimizes an adaptive multi-task loss function in order to jointly predict turn-level Response Quality labels provided by experts and explicit dialogue-level ratings provided by end users. The proposed BiLSTM based deep neural net model automatically weighs each turn’s contribution towards the estimated dialogue-level rating, implicitly encodes tem- poral dependencies, and removes the need to hand-craft features. On dialogues sampled from 28 Alexa domains, two dialogue systems and three user groups, the joint dialogue-level satisfaction estimation model achieved up to an absolute 27% (0.43 → 0.70) and 7% (0.63 → 0.70) improvement in linear correlation performance over baseline deep neural net and benchmark Gradient boost- ing regression models, respectively. 1 Introduction Automatic turn and dialogue level quality evalua- tion of end user interactions with Spoken Dialogue Systems (SDS) is vital for identifying problematic conversations and for optimizing dialogue policy using a data driven approach, such as reinforce- ment learning. One of the main requirements to designing data-driven policies is to automatically and accurately measure the success of an interac- tion. Automated dialogue quality estimation ap- proaches, such as Interaction Quality (IQ) (Schmitt et al., 2012) and recently Response Quality (RQ) ∗Currently at LinkedIn, but did this work at Amazon. (Bodigutla et al., 2019a) were proposed to capture satisfaction at turn level from an end user perspec- tive. Automated models to estimate IQ (Ultes et al., 2014; Schmitt et al., 2011; Asri et al., 2014) used a variety of features derived from the dialogue- turn, dialogue history, and output from three Spo- ken Language Understanding (SLU) components, namely: Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), and the dialogue manager. RQ prediction models (Bod- igutla et al., 2019a) further extended the feature sets with features derived from the dialogue-context, ag- gregate popularity and diversity of topics discussed within a dialogue-session. Using automatically computed diverse feature sets and expert ratings to annotate turns overcame limitations suffered by earlier approaches to mea- sure dialogue quality at turn-level, such as using sparse sentiment signal (Shi and Yu, 2018), intru- sive solicitation of user feedback after each turn, and using manual feature extraction process to es- timate turn-level ratings (Engelbrecht et al., 2009; Higashinaka et al., 2010). For predicting user satisfaction at dialogue-level, IQ estimation approach was shown to generalize to dialogues from different domains (Schmitt and Ultes, 2015). Using annotated user satisfaction rat- ings to estimate dialogue-level quality, overcame the limitation with using …

Content truncated to 500 words