Learning the Human Judgment for the Automatic Evaluation of Chatbot

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1598–1602 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1598 Learning the Human Judgment for the Automatic Evaluation of Chatbot Shih-Hung Wu*, Sheng-Lun Chien Department of Computer Science and Information Engineering, Chaoyang University of Technology, Taichung, Taiwan (R.O.C) {shwu, s10727614}@cyut.edu.tw *Contact-author Abstract It is hard to evaluate the quality of the generated text by a generative dialogue system. Currently, dialogue evaluation relies on human judges to label the quality of the generated text. It is not a reusable mechanism that can give consistent evaluation for system developers. We believe that it is easier to get consistent results on comparing two generated dialogue by two systems and it is hard to give a consistent quality score on only one system at a time. In this paper, we propose a machine learning approach to reduce the effort of human evaluation by learning the human judgment on comparing two dialogue systems. Training from the human labeling result, the evaluation model learns which generative models is better in each dialog context. Thus, it can be used for system developers to compare the fine-tuned models over and over again without the human labor. In our experiment we find the agreement between the learned model and human judge is 70%. The experiment is conducted on comparing two attention based GRU-RNN generative models. Keywords: Chatbot, Dialogue generation, learning for evaluation 1. Introduction Assessment of open domain dialogue systems usually rely on human, and it is hard to have a consistent evaluation. Due to the lack of automatic metrics, there is very limited reproducibility of dialogue system evaluation (Fokkens et al., 2013). At the same time, human evaluation methodologies are also very diverse. Papers report novel generation methods for dialogue systems, but pay little attention on datasets and the evaluation process. Traditional automatic evaluation measures on NLP applications, such as BLEU for machine translation (Papineni et al., 2002), requires human references as ground truth. Recent research on natural language generation makes Chatbot more interesting, however, it is hard to evaluate the quality of the generated dialogue since it is hard to provide human references. Although there are many research on dialogue evaluations (Shawar and Atwell, 2007), most automatic measure metrics cannot reflect the quality of a Chatbot. Currently, the Chatbot evaluation can only rely on human judge, no matter it is for research such as a shared task or for application development (Chen et al., 2019). There are two main reasons on why it is hard to do the automatic evaluation. First at all, it is hard to evaluate natural language of most natural language processing tasks, such as machine translation and summarization. It required predefined reference for the evaluation and golden reference usually do not exist. The second reason is more complicate, since any known automatic evaluation tool can be incorporated into the generation process; the performance will be no different among these systems that incorporated a known automatic …

Content truncated to 500 words