Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining

Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining Ananya B. Sai∗ and Akash Kumar Mohankumar∗ and Siddhartha Arora and Mitesh M. Khapra {ananya, miteshk}@cse.iitm.ac.in, {makashkumar99, sidarora1990}@gmail.com Robert-Bosch Centre for Data Science and Artificial Intelligence Indian Institute of Technology, Madras Abstract There is an increasing focus on model-based dialog evaluation metrics such as ADEM, RUBER, and the more recent BERT-based metrics. These models aim to assign a high score to all relevant responses and a low score to all irrelevant responses. Ideally, such models should be trained using multiple relevant and irrelevant responses for any given context. However, no such data is publicly available, and hence existing models are usually trained using a single relevant response and multiple randomly selected responses from other con- texts (random negatives). To allow for better training and robust evaluation of model-based metrics, we introduce the DailyDialog++ dataset, consisting of (i) five relevant re- sponses for each context and (ii) five adver- sarially crafted irrelevant responses for each context. Using this dataset, we first show that even in the presence of multiple correct refer- ences, n-gram based metrics and embedding based metrics do not perform well at separat- ing relevant responses from even random negatives. While model-based metrics perform better than n-gram and embedding based metrics on random negatives, their perfor- mance drops substantially when evaluated on adversarial examples. To check if large scale pretraining could help, we propose a new BERT-based evaluation metric called DEB, which is pretrained on 727M Reddit conversa- tions and then finetuned on our dataset. DEB significantly outperforms existing models, showing better correlation with human judg- ments and better performance on random negatives (88.27% accuracy). However, its performance again drops substantially when ∗The first two authors worked equally towards the project. evaluated on adversarial responses, thereby highlighting that even large-scale pretrained evaluation models are not robust to the adversarial examples in our dataset. The dataset1 and code2 are publicly available. 1 Introduction Open-domain conversational systems are increas- ingly in demand for several applications ranging from personal digital assistants to entertainers for recreation. While several automated dialogue agents such as Siri, Alexa, Cortana, and Google Assistant have been built and deployed, there is no good automatic evaluation metric to measure the quality of their conversations. Researchers have usually adopted n-gram based metrics (Papineni et al., 2002; Banerjee and Lavie, 2005; Lin, 2004) or embedding based metrics (Forgues et al., 2014; Rus and Lintean, 2012; Zhang et al., 2020a) to compare the model’s response with a single refer- ence. These metrics assume that a valid response should be semantically or lexically similar to the reference without taking the context of the con- versation into consideration. However, in open domain conversations, a given context can have a wide range of possible responses that may be lex- ically and semantically very different from each other. For example, the context, ‘‘I like danc- ing and swimming, what about you?’’ can be responded to with ‘‘I paint in my free time’’ or ‘‘I …

Content truncated to 500 words