DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification

Proceedings of the 29th International Conference on Computational Linguistics, pages 496–504 October 12–17, 2022. 496 Open-Domain Dialog Evaluation using Follow-Ups Likelihood Maxime De Bruyn, Ehsan Lotfi, Jeska Buhmann, Walter Daelemans CLiPS Research Center University of Antwerp, Belgium maxime.debruyn@uantwerpen.be Abstract Automatic evaluation of open-domain dialogs remains an unsolved problem. Moreover, ex- isting methods do not correlate strongly with human annotations. This paper presents a new automated evaluation method using follow-ups: we measure the probability that a language model will continue the conversation with a fixed set of follow-ups (e.g. Not really relevant here, What are you trying to say?). When com- pared against twelve existing methods, our new evaluation achieves the highest correlation with human evaluations. 1 Introduction Despite the recent progress in Natural Language Processing, the automatic evaluation of open- domain conversations remains an unsolved prob- lem. It is difficult to establish criteria to measure the quality of a system. Task-oriented dialog sys- tems use metrics such as task success or dialog efficiency. However, these do not apply to open- domain conversational agents (McTear, 2020). Currently, there are two options for open-domain dialog evaluation: human evaluation and automated evaluation. Thanks to their understanding of natu- ral language, humans are able to digest the entire dialog context in order to meaningfully evaluate a response (Mehri et al., 2022). Human evaluation also has its shortcomings: inconsistency in ratings (the same annotator may give two different scores depending on the mood), lack of reproducibility, and cost (Mehri et al., 2022). The second option is to use automated evalua- tion metrics. Methods inherited from sequence-to- sequence machine translation such as BLEU (Pap- ineni et al., 2002) evaluate the generated utterance by comparing it to the ground-truth. By doing so, these methods miss the one-to-many characteristic of conversation: a conversation may evolve in more than one valid direction. Figure 1: Illustration of our method. We measure the probability (log-likelihood) that a language model will continue the conversation with a set of predefined follow-ups. This paper shows that the sum of the indi- vidual log-likelihoods correlates strongly with human evaluations. To tackle this problem, researchers came up with reference-free evaluation metrics: the generated utterance is not compared to a ground truth but evaluated on its own. FED (Mehri and Eskenazi, 2020a) is an unsu- pervised reference-free evaluation metric. It uses the idea that one can use the next utterance in a conversation to rate the turn before it. When users speak to a system, their response to a given system may implicitly provide feedback for the system. FED uses a set of predefined follow-ups and the log-likelihood from a language model to measure 18 fine-grained attributes in a conversation. Inspired by the FED metric, we propose a new evaluation method called FULL (Follow-Up Log- Likelihood). We start by explaining our method and how it departs from the original FED metric. Next, we explain our choice of language model and follow-ups. Finally, we demonstrate that our new method achieves the highest correlation with hu- man evaluations compared …

Content truncated to 500 words