The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2071–2082 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2071 The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues Alberto Testoni DISI - University of Trento Trento - Italy alberto.testoni@unitn.it Raffaella Bernardi CIMeC and DISI - University of Trento Rovereto (TN) - Italy raffaella.bernardi@unitn.it Abstract When training a model on referential dialogue guessing games, the best model is usually cho- sen based on its task success. We show that in the popular end-to-end approach, this choice prevents the model from learning to generate linguistically richer dialogues, since the ac- quisition of language proficiency takes longer than learning the guessing task. By compar- ing models playing different games (Guess- What, GuessWhich, and Mutual Friends), we show that this discrepancy is model- and task- agnostic. We investigate whether and when better language quality could lead to higher task success. We show that in GuessWhat, models could increase their accuracy if they learn to ground, encode, and decode also words that do not occur frequently in the train- ing set. 1 Introduction A good dialogue model should generate utterances that are indistinguishable from human dialogues (Liu et al., 2016; Li et al., 2017). This holds for both chit-chat, open-domain, and task-oriented dialogues. While chit-chat dialogue systems are usually evaluated by analysing the quality of their dialogues (Lowe et al., 2017; See et al., 2019), task-oriented dialogue models are evaluated on their task success and it is common practice to choose the best model based only on the task suc- cess metric. We explore whether this choice pre- vents the system from learning better linguistic skills. Important progress has been made on the devel- opment of such conversational agents. The boost is mostly due to the introduction of the encoder- decoder framework (Sutskever et al., 2014) which allows learning directly from raw data to both un- derstand and generate utterances. The framework has been found to be promising both for chit- chat (Vinyals and Le, 2015) and task-oriented di- alogues (Lewis et al., 2017), and it has been fur- ther extended to develop agents that can commu- nicate through natural language about visual con- tent (Mostafazadeh et al., 2017; Das et al., 2017a; de Vries et al., 2017). Several dialogue tasks have been proposed as referential guessing games in which an agent (the Q-bot) asks questions to an- other agent (the A-bot) and has to guess the refer- ent (e.g., a specific object depicted in the image) they have been speaking about (de Vries et al., 2017; Das et al., 2017b; He et al., 2017; Haber et al., 2019; Ilinykh et al., 2019; Udagawa and Aizawa, 2019). We are interested in understand- ing the interplay between the learning processes behind these two sub-tasks: generating questions and guessing the referent. Shekhar et al. (2019) have compared models on GuessWhat and have shown that task suc- cess (TS) does not correlate with …

Content truncated to 500 words