Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4053–4064 June 6–11, 2021. ©2021 Association for Computational Linguistics 4053 Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents Mohammad Kachuee∗ UCLA Computer Science Los Angeles, CA mkachuee@cs.ucla.edu Hao Yuan Amazon Alexa AI Seattle, WA Young-Bum Kim Amazon Alexa AI Seattle, WA {yuanha,youngbum,sungjinl}@amazon.com Sungjin Lee Amazon Alexa AI Seattle, WA Abstract Turn-level user satisfaction is one of the most important performance metrics for conversa- tional agents. It can be used to monitor the agent’s performance and provide insights about defective user experiences. While end- to-end deep learning has shown promising re- sults, having access to a large number of reli- able annotated samples required by these meth- ods remains challenging. In a large-scale con- versational system, there is a growing number of newly developed skills, making the tradi- tional data collection, annotation, and model- ing process impractical due to the required an- notation costs and the turnaround times. In this paper, we suggest a self-supervised con- trastive learning approach that leverages the pool of unlabeled data to learn user-agent inter- actions. We show that the pre-trained models using the self-supervised objective are transfer- able to the user satisfaction prediction. In ad- dition, we propose a novel few-shot transfer learning approach that ensures better transfer- ability for very small sample sizes. The sug- gested few-shot method does not require any inner loop optimization process and is scal- able to very large datasets and complex mod- els. Based on our experiments using real data from a large-scale commercial system, the sug- gested approach is able to significantly reduce the required number of annotations, while im- proving the generalization on unseen skills. 1 Introduction Nowadays automated conversational agents such as Alexa, Siri, Google Assistant, Cortana, etc. are widespread and play an important role in many dif- ferent aspects of our lives. Their applications vary from storytelling and education for children to as- sisting the elderly and disabled with their daily activities. Any successful conversational agent should be able to communicate in different lan- guages and accents, understand the conversation ∗Work done as an intern at Amazon Alexa AI. context, analyze the query paraphrases, and route the requests to various skills available for handling the user’s request (Ram et al., 2018). In such a large-scale system with many compo- nents, it is crucial to understand if the human user is satisfied with the automated agent’s response and actions. In other words, it is desirable to know if the agent is communicating properly and providing the service that is expected by the user. In the literature, it is referred to as targeted turn-level satisfaction as we are only interested in the user’s satisfaction for a certain conversation turn given the context of the conversation, and not the overall satisfaction for the whole conversation (Park et al., 2020). Perhaps the most basic use of a user satisfaction model would be to monitor the performance …

Content truncated to 500 words