Language Model Transformers as Evaluators for Open-domain Dialogues

Proceedings of the 28th International Conference on Computational Linguistics, pages 6797–6808 Barcelona, Spain (Online), December 8-13, 2020 6797 Language Model Transformers as Evaluators for Open-domain Dialogues Rostislav Nedelchev1 Jens Lehmann1,2 Ricardo Usbeck1,2 1Smart Data Analytics Group, University of Bonn, Germany 2Fraunhofer IAIS, Sankt Augustin and Dresden, Germany rostislav.nedelchev@uni-bonn.de, jens.lehmann@cs.uni-bonn.de ricardo.usbeck@iais.fraunhofer.de Abstract Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. So far, the most effective way to assess the quality of the dialogues produced by these systems is to use resource-intensive manual labor instead of automated means. In this work, we investigate whether language models (LM) based on transformer neural networks can indicate the quality of a conversation. In a general sense, language models are methods that learn to predict one or more words based on an already given context. Due to their unsupervised nature, they are candidates for efficient, automatic indication of dialogue quality. We demonstrate that human evaluators have a positive correlation between the output of the language models and scores. We also provide some insights into their behavior and inner-working in a conversational context. 1 Introduction Lately, deep learning conversational systems have seen increasing interest from industry and academia alike (Chen et al., 2017). These systems find usage in various contexts, starting from personal speech assistants like Google Assistant through the ”chatbots” on instant messaging platforms like Facebook Messenger, and finally, conversational services like LUIS1. Many of these applications serve the objective of completing a specific function like purchasing a product or booking services (e.g., hotels, flights). Nonetheless, these applications can still profit from open-domain dialogue skills like chit-chatting, which would provide a more human-like interaction with users. Presently, scientists and engineers working on computer-based conversational systems need human- based evaluation to assess the quality and usability of their work (Dinan et al., 2019; Logacheva et al., 2018; Yoshino et al., 2019). These evaluations are costly in terms of resources. Thus, the field of dialogue systems could take advantage of an automated method for assessing conversations. Seminal works in text summarization and machine translation have already proposed their field- specific metrics for automated assessments - for the former ROUGE (Lin, 2004), and, for the latter, BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005). Dialogue system research (Rit- ter et al., 2011; Serban et al., 2016; Yoshino et al., 2019) constantly uses these metrics. However, Liu et al. show that these metrics based on word-overlap between prediction and references are not reliable for evaluating the usefulness of dialogue systems (2016). Hence, the field should use more sophisticated methods that consider the previous utterances of a conversation and its semantic meaning. When human annotators evaluate a dialogue, they do not use an explicit reference or necessarily seek word overlap between context and response (or the lack of it). Their assessment bases itself on experience with the language and the implicit knowledge they have about it. The core principle of statistical language models (LM) is to capture and reproduce these properties. …

Content truncated to 500 words