Proxy Indicators for the Quality of Open-domain Dialogues

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7834–7855 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7834 Proxy Indicators for the Quality of Open-domain Dialogues Rostislav Nedelchev Smart Data Analytics Group, University of Bonn, Germany rostislav.nedelchev@uni-bonn.de Jens Lehmann Smart Data Analytics Group, University of Bonn, Germany Fraunhofer IAIS, Sankt Augustin and Dresden, Germany jens.lehmann@cs.uni-bonn.de Ricardo Usbeck Semantic Systems Group, University of Hamburg, Germany Smart Data Analytics Group, University of Bonn, Germany ricardo.usbeck@uni-hamburg.de Abstract The automatic evaluation of open-domain di- alogues remains a largely unsolved challenge. Thus, despite the abundance of work done in the field, human judges have to evaluate dia- logues’ quality. As a consequence, performing such evaluations at scale is usually expensive. This work investigates using a deep-learning model trained on the General Language Un- derstanding Evaluation (GLUE) benchmark to serve as a quality indication of open-domain dialogues. The aim is to use the various GLUE tasks as different perspectives on judging the quality of conversation, thus reducing the need for additional training data or responses that serve as quality references. Due to this nature, the method can infer various quality metrics and derive a component-based overall score. We achieve statistically significant correlation coefficients of up to 0.7. 1 Introduction Recently, dialogue systems powered by machine learning have gathered much attention from indus- try and academia alike (Chen et al., 2017). These systems have various applications, such as per- sonal speech assistants, customer service, techni- cal support, and training and education. In most cases, these systems are task-specific and help with tasks like booking a restaurant. Nevertheless, they can still benefit from open-domain conversational skills, such as the ability to chit-chat to enable natural dialogues, rather than repeating the input utterance like a parrot. Nowadays, people working in this field have to use human annotators to evaluate the quality of a conversation (Dinan et al., 2019; Logacheva et al., 2018; Yoshino et al., 2019), which can be very costly in terms of resources. Thus, these sys- tems’ research and development could benefit sig- nificantly from an automated approach to evaluate conversations. Research in the related fields of text summa- rization and machine translation has developed au- tomated measures for evaluation. Some notable examples are, for the former, ROUGE (Lin, 2004) and, for the latter, BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005). These are also adopted by works researching dialogue systems (Ritter et al., 2011; Serban et al., 2016; Yoshino et al., 2019). However, Liu et al. (2016) demonstrated that these metrics are not suitable for replacing human evaluators. Also, Sai et al. (2020) reported that even using multiple instead of single references and an overlap-based approach still un- derperforms. Thus, more advanced techniques are needed that consider the context and semantics of a dialogue. Human annotators can distinguish bad from good quality dialogues from an intuitive perspec- tive, not because they necessarily have been taught to do so. Instead, people have a notion of a fluent text or …

Content truncated to 500 words