Are we Estimating or Guesstimating Translation Quality?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 593–599 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 593 Evaluating Dialogue Generation Systems via Response Selection Shiki Sato1 Reina Akama1,2 Hiroki Ouchi2,1 Jun Suzuki1,2 Kentaro Inui1,2 1Tohoku University 2RIKEN {shiki.sato,reina.a,jun.suzuki,inui}@ecei.tohoku.ac.jp hiroki.ouchi@riken.jp Abstract Existing automatic evaluation metrics for open-domain dialogue response generation systems correlate poorly with human evalua- tion. We focus on evaluating response gener- ation systems via response selection. To eval- uate systems properly via response selection, we propose a method to construct response se- lection test sets with well-chosen false candi- dates. Specifically, we propose to construct test sets filtering out some types of false can- didates: (i) those unrelated to the ground-truth response and (ii) those acceptable as appro- priate responses. Through experiments, we demonstrate that evaluating systems via re- sponse selection with the test set developed by our method correlates more strongly with hu- man evaluation, compared with widely used automatic evaluation metrics such as BLEU. 1 Introduction Automatic evaluation for open-domain dialogue generation systems has a potential for driving their research and development because of its high re- producibility and low cost. However, existing auto- matic evaluation metrics, such as BLEU (Papineni et al., 2002), correlate poorly with human evalua- tion (Liu et al., 2016). This poor correlation arises from a nature of dialogue, that is, there are many acceptable responses to an input context, known as the one-to-many problem (Zhao et al., 2017). To tackle this problematic issue, we focus on evaluating response generation systems via re- sponse selection. In this task, systems select an appropriate response for a given context from a set of response candidates. Each candidate has the label that indicates whether the candidate is appropriate response for the given context. Tradi- tionally, response selection has been used to evalu- ate retrieval-based dialogue systems (Lowe et al., 2015; Wu et al., 2017). We consider applying this task to driving the research for dialogue generation Repository Context: Do you have a car? Ground-Truth: Yes, I have a car. Query No, I have a car. I don’t know. I have a cold. No, I have a car. I don’t know. I have a cold. 12 4 Question Retrieve utterances Give scores by human evaluation Remove high-score utterances No, I have a car. I don’t know. I have a cold. 2 1 4 False candidates Figure 1: Overview of the construction method of our test set. First, we retrieve only utterances related to the ground-truth response from a repository. Then, we remove acceptable utterances by human evaluation. systems. Specifically, we consider using response selection to pick out promising systems that should be evaluated more precisely by humans among a lot of candidate systems. We assume that response selection is a valid option for such a preliminary evaluation on the basis of the following assumption: systems that can generate appropriate responses can also select appropriate responses. One advan- tage of evaluating generation systems via response selection is that it can …

Content truncated to 500 words