Predicting Ratings of Real Dialogue Participants from Artificial Data and Ratings of Human Dialogue Observers

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 726–734 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 726 Predicting Ratings of Real Dialogue Participants from Artificial Data and Ratings of Human Dialogue Observers Kallirroi Georgila, Carla Gordon, Volodymyr Yanov, David Traum Institute for Creative Technologies, University of Southern California 12015 Waterfront Drive, Los Angeles, CA 90094-2536, USA {kgeorgila, cgordon, yanov, traum}@ict.usc.edu Abstract We collected a corpus of dialogues in a Wizard of Oz (WOz) setting in the Internet of Things (IoT) domain. We asked users participating in these dialogues to rate the system on a number of aspects, namely, intelligence, naturalness, personality, friendliness, their enjoyment, overall quality, and whether they would recommend the system to others. Then we asked dialogue observers, i.e., Amazon Mechanical Turkers (MTurkers), to rate these dialogues on the same aspects. We also generated simulated dialogues between dialogue policies and simulated users and asked MTurkers to rate them again on the same aspects. Using linear regression, we developed dialogue evaluation functions based on features from the simulated dialogues and the MTurkers’ ratings, the WOz dialogues and the MTurkers’ ratings, and the WOz dialogues and the WOz participants’ ratings. We applied all these dialogue evaluation functions to a held-out portion of our WOz dialogues, and we report results on the predictive power of these different types of dialogue evaluation functions. Our results suggest that for three conversational aspects (intelligence, naturalness, overall quality) just training evaluation functions on simulated data could be sufficient. Keywords: dialogue evaluation functions, real and simulated dialogues, Internet of Things 1. Introduction In order to build a dialogue system for a particular domain, data in this domain are required. Ideally these data should record interactions between real users and a dialogue sys- tem, or, if a dialogue system is not available (which is very common in the initial stages of development), interactions between real users and a Wizard (a human playing the role of the system), in a so called Wizard of Oz (WOz) set- ting (Dahlb¨ack et al., 1993). However, this approach can be quite expensive and time consuming. Building the ini- tial dialogue system or the WOz environment can be costly. Furthermore, training one or more Wizards and recruiting human participants to interact with the Wizards can signifi- cantly add to the overall cost, and the whole process can be time consuming. An alternative cheaper approach is to gen- erate artificial data, either by having dialogue policies inter- act with simulated users and generate a very large number of dialogues (El Asri et al., 2016), or by having linguists write a variety of dialogues (Georgila et al., 2018). Here we use human-system data collected in a WOz setting, and simulated dialogues generated by having dialogue poli- cies interact with simulated users. Simulated dialogues can be very useful in bootstrapping the development of a dia- logue system, but they do not fully represent how real users would react in real time to system interventions …

Content truncated to 500 words