Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4079–4090 August 1–6, 2021. ©2021 Association for Computational Linguistics 4079 Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentation Varun Gangal ∗1 Harsh Jhamtani * 1 Eduard Hovy 1 Taylor Berg-Kirkpatrick 2 1 School of Computer Science, Carnegie Mellon University 2 Computer Science and Engineering. University of California San Diego {vgangal,jharsh,hovy}@cs.cmu.edu, tberg@ucsd.eng.edu Abstract Multiple different responses are often plau- sible for a given open domain dialog con- text. Prior work has shown the importance of having multiple valid reference responses for meaningful and robust automated evaluations. In such cases, common practice has been to collect more human written references. How- ever, such collection can be expensive, time consuming, and not easily scalable. Instead, we propose a novel technique for automati- cally expanding a human generated reference to a set of candidate references. We fetch plausible references from knowledge sources, and adapt them so that they are more flu- ent in context of the dialog instance in ques- tion. More specifically, we use (1) a common- sense knowledge base to elicit a large num- ber of plausible reactions given the dialog his- tory (2) relevant instances retrieved from dia- log corpus, using similar past as well as future contexts. We demonstrate that our automati- cally expanded reference sets lead to large im- provements in correlations of automated met- rics with human ratings of system outputs for DailyDialog dataset. 1 1 Introduction Evaluation by human annotators perhaps give the best insights into quality of machine generated nat- ural language outputs. However, they can be expen- sive and time consuming. Much focus has therefore been on automated evaluation methods which cor- relate with human evaluations. Automated metrics such as BLEU (Papineni et al., 2002) work well for tasks such as machine translation, but often cor- relate poorly with human ratings in tasks such as open domain dialog which admit a wide variety of ∗ VG and HJ contributed equally for this paper. Order decided by coin flip. 1Code and data are available at https://github.com/harsh19/ Diverse-Reference-Augmentation/ Figure 1: We propose automatic ways to collect references sans any crowd-sourcing, through two types of knowledge sources: commonsense and retrieved instance knowledge, fol- lowed by automated adaptation to make them more fluent in the target contexts. valid response for given context, often due to small number of human written references (Zhao et al., 2017; Sai et al., 2020b). Prior work (Sugiyama et al., 2019; Gupta et al., 2019) has demonstrated that having multiple valid references for the same context leads to automated metrics being better cor- related to human judgements for appropriateness. However, collecting human written responses is difficult to scale, can be costly, and may find it dif- ficult to cover a large variety of correct responses (Celikyilmaz et al., 2020). In this work, we automatically extract a large number of diverse references to be used with such reference-based metrics, without resorting to ex- pensive crowd-sourcing. Intuitively, since open- domain dialog pertains to everyday …

Content truncated to 500 words