Achieving Reliable Human Assessment of Open-Domain Dialogue Systems

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6416 - 6437 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Achieving Reliable Human Assessment of Open-Domain Dialogue Systems Tianbo Ji1,2, Yvette Graham1,3, Gareth Jones1,2, Chenyang Lyu2, and Qun Liu4 1ADAPT Centre 2School of Computing, Dublin City University 3School of Computer Science and Statistics, Trinity College Dublin 4Noah’s Ark Lab, Huawei {tianbo.ji,yvette.graham,gareth.jones}@ adaptcentre.ie, chenyang.lyu2@mail.dcu.ie, qun.liu@huawei.com Abstract Evaluation of open-domain dialogue systems is highly challenging and development of bet- ter techniques is highlighted time and again as desperately needed. Despite substantial efforts to carry out reliable live evaluation of systems in recent competitions, annotations have been abandoned and reported as too unreliable to yield sensible results. This is a serious prob- lem since automatic metrics are not known to provide a good indication of what may or may not be a high-quality conversation. Answering the distress call of competitions that have em- phasized the urgent need for better evaluation techniques in dialogue, we present the success- ful development of human evaluation that is highly reliable while still remaining feasible and low cost. Self-replication experiments re- veal almost perfectly repeatable results with a correlation of r = 0.969. Furthermore, due to the lack of appropriate methods of statistical significance testing, the likelihood of poten- tial improvements to systems occurring due to chance is rarely taken into account in dialogue evaluation, and the evaluation we propose fa- cilitates application of standard tests. Since we have developed a highly reliable evaluation method, new insights into system performance can be revealed. We therefore include a com- parison of state-of-the-art models (i) with and without personas, to measure the contribution of personas to conversation quality, as well as (ii) prescribed versus freely chosen topics. In- terestingly with respect to personas, results in- dicate that personas do not positively contribute to conversation quality as expected. 1 Introduction Evaluation of open-domain dialogue is particularly challenging and has been cited in high-profile com- petitions as a known open problem (Dinan et al., 2019). Challenges arise primarily from the fact that in real-world conversations there exists such a vast number of possible appropriate responses. Subsequently, dialogue evaluation that relies on comparison with pre-created reference dialogues incur substantial false-negative rates as many ap- propriate responses are unfairly penalized simply for not corresponding closely with references. In addition, evaluation faces further challenges with respect to the ability to fully take into account dia- logue history.1 In this paper, we present a new method of open- domain dialogue evaluation based on human as- sessment of live conversations with models that avoids the need for pre-created reference dialogues and ensures full familiarity with dialogue history, ticking two important boxes in terms of validity. Although live human evaluation of models has the advantage of being highly valid, reliability unfortu- nately cannot be assumed and developing methods of evaluation for language tasks that achieve high rater consistency has been challenging, often result- ing in low levels of agreement between …

Content truncated to 500 words