HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3652–3665 August 1–6, 2021. ©2021 Association for Computational Linguistics 3652 HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations Weixin Liang1 Stanford University wxliang@stanford.edu Kai-Hui Liang1 Columbia University kaihui.liang@columbia.edu Zhou Yu Columbia University zy2461@columbia.edu Abstract Open-domain dialog systems have a user- centric goal: to provide humans with an en- gaging conversation experience. User engage- ment is one of the most important metrics for evaluating open-domain dialog systems, and could also be used as real-time feedback to benefit dialog policy learning. Existing work on detecting user disengagement typi- cally requires hand-labeling many dialog sam- ples. We propose HERALD, an efficient an- notation framework that reframes the training data annotation process as a denoising prob- lem. Specifically, instead of manually label- ing training samples, we first use a set of la- beling heuristics to label training samples au- tomatically. We then denoise the weakly la- beled data using the Shapley algorithm. Fi- nally, we use the denoised data to train a user engagement detector. Our experiments show that HERALD improves annotation efficiency significantly and achieves 86% user disengage- ment detection accuracy in two dialog corpora. Our implementation is available at https:// github.com/Weixin-Liang/HERALD/. 1 Introduction Evaluation metrics heavily influence a field’s re- search direction. The ultimate goal of open-domain dialog systems is to provide an enjoyable experi- ence to users. Previous research mainly focuses on optimizing automatic dialog evaluation metrics such as BLEU, which models the distance between the system responses and a limited number of ref- erences available. However, it has been shown that these metrics correlate poorly with human judg- ments (Liu et al., 2016). Open-domain dialog system evaluation has long been one of the most difficult challenges in the dia- log community for several reasons: (1) The goal of 1Equal Contribution. dialog evaluation should be to evaluate users’ con- versational experience. Existing automatic evalua- tion metrics such as BLEU are mostly constrained to a static corpus, and do not capture the user experi- ence in a realistic interactive setting. (2) Currently, self-reported user ratings are widely used to evalu- ate open-domain dialogs. However, self-reported ratings suffer from bias and variance among differ- ent users (Liang et al., 2020e). Although we could tell which dialog system is better by running statis- tical tests on a large number of noisy ratings, it is challenging to locate dialogs with bad performance reliably. Only by identifying these bad dialogs ef- fectively can we correct errors in these samples to improve dialog system quality. User engagement has been recognized as one of the essential metrics for open-domain dialog evaluation (Ram et al., 2018). Previous research also confirms that incorporating user engagement as real-time feedback benefits dialog policy learn- ing (Yu et al., 2016). One of the most costly bot- tlenecks of learning to detect user disengagement is to annotate many turn-level user engagement la- bels (Ghazarian et al., 2020). …

Content truncated to 500 words