Speaker Sensitive Response Evaluation Model

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6376–6385 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6376 Speaker Sensitive Response Evaluation Model JinYeong Bak School of Computing KAIST jy.bak@kaist.ac.kr Alice Oh School of Computing KAIST alice.oh@kaist.edu Abstract Automatic evaluation of open-domain dia- logue response generation is very challenging because there are many appropriate responses for a given context. Existing evaluation models merely compare the generated response with the ground truth response and rate many of the appropriate responses as inappropriate if they deviate from the ground truth. One approach to resolve this problem is to consider the sim- ilarity of the generated response with the con- versational context. In this paper, we propose an automatic evaluation model based on that idea and learn the model parameters from an unlabeled conversation corpus. Our approach considers the speakers in defining the differ- ent levels of similar context. We use a Twitter conversation corpus that contains many speak- ers and conversations to test our evaluation model. Experiments show that our model out- performs the other existing evaluation metrics in terms of high correlation with human an- notation scores. We also show that our model trained on Twitter can be applied to movie dia- logues without any additional training. We pro- vide our code and the learned parameters so that they can be used for automatic evaluation of dialogue response generation models. 1 Introduction Evaluating the system generated responses for open-domain dialogue is a difficult task. There are many possible appropriate responses given a dia- logue context, and automatic metrics such as BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) rate the responses that deviate from the ground truth as inappropriate. Still, it is important to develop and use an automatic metric because human an- notation is very costly. In addition to BLEU and ROUGE, there is a widely-used evaluation metric based on the distributed word representation (Liu et al., 2016), but this metric shows low correlations with human judgments. One reason for the difficulty in developing an automatic metric that correlates well with human judgements is that the range of appropriate re- sponses for a given context is very wide. Table 1 shows an example of a conversation between Speaker A and B. While there is a ground truth re- sponse “Yeah let’s go to the theater,” A could have also said “That sounds good! Have you seen Thor?” or “Good. What movie?” Note that based on word overlap with the ground truth, these two responses would receive low scores. Responses labeled N#, such as “The weather is no good for walking” are not appropriate. As the Table shows, the existing metrics from BLEU to RUBER are not able to tell apart these appropriate A# responses from the in- approriate N# responses. Some recent metrics such as ADEM (Lowe et al., 2017) and RUBER (Tao et al., 2018) com- pute the similarity between a context and a gener- ated response. However, ADEM requires human- …

Content truncated to 500 words