Classical Sequence Match Is a Competitive Few-Shot One-Class Learner

Proceedings of the 29th International Conference on Computational Linguistics, pages 505–509 October 12–17, 2022. 505 Joint Goal Segmentation and Goal Success Prediction on Multi-Domain Conversations Meiguo Wang Benjamin Yao Bin Guo Xiaohu Liu Yu Zhang Tuan-Hung Pham Chenlei Guo Amazon Alexa AI Abstract To evaluate the performance of a multi-domain goal-oriented Dialogue System (DS), it is im- portant to understand what the users’ goals are for the conversations and whether those goals are successfully achieved. The success rate of goals directly correlates with user satisfaction and perceived usefulness of the DS. In this pa- per, we propose a novel automatic dialogue evaluation framework that jointly performs two tasks: goal segmentation and goal success prediction. We extend the RoBERTa-IQ model (Gupta et al., 2021) by adding multi-task learn- ing heads for goal segmentation and success prediction. Using an annotated dataset from a commercial DS, we demonstrate that our pro- posed model reaches an accuracy that is on-par with single-pass human annotation comparing to a three-pass gold annotation benchmark. 1 Introduction Today, commercial conversational AI assistants (e.g., Amazon Alexa, Apple Siri, and Google As- sistant) are increasingly popular. However, it is challenging to establish reliable metrics to continu- ously measure the system performance in business reports and A/B experiments. The commonly used metrics in industry, e.g., monthly active users, di- alogue count per user, and downstream impacts such as increased subscriptions and product sales etc., usually move slowly and are not suitable to measure the impact of functionality changes. On the other hand, the dissatisfaction metrics based on manual annotation, which can capture dissatisfying user experiences due to system errors, incomplete service coverage, or poor response quality etc., are sensitive to functionality changes, but not suitable for online monitoring and experimentation due to their offline nature. Automatic dialogue evaluation metrics (Schmitt and Ultes, 2015; Ling et al., 2020) aim to combine the benefits of the two types of met- rics mentioned above by providing an online metric Figure 1: A dialogue that contains two goals: play- music and turn-on-device. Here, a turn is defined as one back-and-forth between the user and agent. A dia- logue is a set of turns in quick succession as indicated by the timestamps. A goal represents an objective the user is trying to accomplish in a sequence of turns. Hu- man annotation labels for turns (defect/non-defect) and goals (success/failure) are provided for reference. that measures user experience and is sensitive to functionality changes. The effectiveness of such metrics depends on how well they are correlated with the human annotations. Automated dialogue evaluation metrics in the literature could be generally grouped into two cate- gories (Deriu et al., 2020): 1) Interaction Quality (IQ) metrics (e.g., (Schmitt and Ultes, 2015; Ling et al., 2020; Gupta et al., 2021)) that measures user experiences at turn level; and 2) Dialogue Quality (DQ) metrics (e.g., (Sun et al., 2021)) that measures user experiences for the whole dialogue. However, we argue that, for multi-domain goal-oriented dia- logue systems, there is a need to …

Content truncated to 500 words