On Quality Ratings for Spoken Dialogue Systems – Experts vs. Users

Proceedings of NAACL-HLT 2013, pages 569–578, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics On Quality Ratings for Spoken Dialogue Systems – Experts vs. Users Stefan Ultes, Alexander Schmitt, and Wolfgang Minker Ulm University Albert-Einstein-Allee 43 89073 Ulm, Germany {stefan.ultes,alexander.schmitt,wolfgang.minker}@uni-ulm.de Abstract In the field of Intelligent User Interfaces, Spo- ken Dialogue Systems (SDSs) play a key role as speech represents a true intuitive means of human communication. Deriving informa- tion about its quality can help rendering SDSs more user-adaptive. Work on automatic esti- mation of subjective quality usually relies on statistical models. To create those, manual data annotation is required, which may be per- formed by actual users or by experts. Here, both variants have their advantages and draw- backs. In this paper, we analyze the relation- ship between user and expert ratings by in- vestigating models which combine the advan- tages of both types of ratings. We explore two novel approaches using statistical classifica- tion methods and evaluate those with a pre- existing corpus providing user and expert rat- ings. After analyzing the results, we eventu- ally recommend to use expert ratings instead of user ratings in general. 1 Introduction and Motivation In human-machine interaction it is important that user interfaces can adapt to the specific requirements of its users. Handicapped persons or angry users, for example, have specific needs and should be treated differently than regular users. Speech is a major component of modern user in- terfaces as it is the natural means of human com- munication. Therefore, it seems logical to use Spo- ken Dialogue Systems (SDS) as part of Intelligent User Interfaces enabling speech communication of different complexity reaching from simple spoken commands up to complex dialogues. Besides the spoken words, the speech signal also may be used to acquire information about the user state, e.g., about their emotional state (cf., e.g., (Polzehl et al., 2011))). By additional analysis of the human- computer-dialogues, even more abstract informa- tion may be derived, e.g., the quality of the system (cf., e.g., (Engelbrecht and M¨oller, 2010)). System quality information may be used to adapt the sys- tem’s behavior online during the ongoing dialogue (cf. (Ultes et al., 2012)). For determining the quality of Spoken Dialogue Systems, several aspects are of interest. M¨oller et al. (2009) presented a taxonomy of quality criteria. They describe quality as a bipartite issue consisting of Quality of Service (QoS) and Quality of Experi- ence (QoE). Quality of Service describes objective criteria like dialogue duration or number of turns. While these are well-defined items that can be de- termined easily, Quality of Experience, which de- scribes the user experience with subjective criteria, is more vague and without a sound definition, e.g., User Satisfaction (US). Subjective aspects like US are either determined by using questionnaires like SASSI (Hone and Gra- ham, 2000) or the ITU-standard augmented frame- work for questionnaires (M¨oller, 2003), or by us- ing single-valued ratings, i.e., a rater only applies one single score. In general, two major categories of work on …

Content truncated to 500 words