Generating Negative Samples by Manipulating Golden Responses for Unsupervised Learning of a Response Evaluation Model

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1525–1534 June 6–11, 2021. ©2021 Association for Computational Linguistics 1525 Generating Negative Samples by Manipulating Golden Responses for Unsupervised Learning of a Response Evaluation Model ChaeHun Park Eugene Jang Wonsuk Yang Jong C. Park∗ School of Computing Korea Advanced Institute of Science and Technology {ddehun,eugenej,derrick0511,park}@nlp.kaist.ac.kr Abstract Evaluating the quality of responses generated by open-domain conversation systems is a challenging task. This is partly because there can be multiple appropriate responses to a given dialogue history. Reference-based met- rics that rely on comparisons to a set of known correct responses often fail to account for this variety, and consequently correlate poorly with human judgment. To address this problem, re- searchers have investigated the possibility of assessing response quality without using a set of known correct responses. Tao et al. (2018) demonstrated that an automatic response eval- uation model could be made using unsuper- vised learning for the next-utterance prediction (NUP) task. For unsupervised learning of such a model, we propose a method of manipulat- ing a golden response to create a new negative response that is designed to be inappropriate within the context while maintaining high sim- ilarity with the original golden response. We find, from our experiments on English datasets, that using the negative samples generated by our method alongside random negative sam- ples can increase the model’s correlation with human evaluations. The process of generating such negative samples is automated and does not rely on human annotation.1 1 Introduction Automatic evaluation of responses can be difficult because multiple answers could be suitable for a single context. Well-known metrics often used in machine translation or text summarization, such as BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), or ROUGE (Lin, 2004), are based on measuring n-gram overlap with a set of human- annotated golden answers. Compared to machine ∗ Corresponding author 1The code is available at https://github.com/ nlpcl-lab/dialog-eval-hard-negative. Figure 1: Example of the three different types of re- sponses for a given dialogue history. Our method manipulates the original response "What’s wrong with heading out with Mark for vacation?" to generate the negative sample "What’s wrong? Go out with Mark for dinner." translation or text summarization systems, conver- sational systems have a wider range of acceptable responses to a given situation (dialogue history). This could explain the low correlation between n-gram-based evaluations and human-conducted evaluations for responses generated by conversa- tion systems, as reported by Liu et al. (2016). They also suggested calculating the embedding similar- ities between responses and correct answers, and showed that these metrics had a higher correlation with human evaluations than n-gram-based metrics. As this method only rewards responses similar to ones in the fixed set of answer candidates, however, it still fails to account for other possible answers that are dissimilar to the known answers. To solve this problem, Lowe et al. (2017) pro- posed a supervised regression model that makes predictions independent …

Content truncated to 500 words