The SpeDial datasets: datasets for Spoken Dialogue Systems analytics

The SpeDial Datasets: Datasets for Spoken Dialogue Systems Analytics Jos´e Lopes1, Arodami Chorianopoulou2 Elisavet Palogiannidi2, Helena Moniz3, Alberto Abad3 Katerina Louka4, Elias Iosif 5,6, Alexandros Potamianos5,6 1 KTH Speech, Music and Hearing, Stockholm, Sweden 2 School of ECE, Technical University of Crete, Greece 3 INESC-ID, Lisboa, Portugal 4 Voiceweb S.A., Greece 5“Athena” Research and Innovation Center, Greece 6School of ECE, National Technical University of Athens, Greece jdlopes@kth.se Abstract The SpeDial consortium is sharing two datasets that were used during the SpeDial project. By sharing them with the community we are providing a resource to reduce the duration of cycle of development of new Spoken Dialogue Systems (SDSs). The datasets include audios and several manual annotations, i.e., miscommunication, anger, satisfaction, repetition, gender and task success. The datasets were created with data from real users and cover two different languages: English and Greek. Detectors for miscommunication, anger and gender were trained for both systems. The detectors were particularly accurate in tasks where humans have high annotator agreement such as miscommunication and gender. As expected due to the subjectivity of the task, the anger detector had a less satisfactory performance. Nevertheless, we proved that the automatic detection of situations that can lead to problems in SDSs is possible and can be a promising direction to reduce the duration of SDS’s development cycle. Keywords: Spoken Dialogue Systems, Multi-lingual Data, Emotions, Sentiment Analysis 1. Introduction The speech services industry has been growing both for telephony applications and, recently, also for smartphones (e.g., Siri). Despite recent progress in Spoken Dia- logue System (SDS) technologies the development cycle of speech services still requires significant effort, expertise and time. Developers often have to wait until the system goes live to detect potential sources of errors that could not be predicted in the design phase. The SpeDial con- sortium (www.spedial.eu) worked to create a semi- automated process for spoken dialogue service develop- ment and speech service enhancement of deployed services. Our main goal was to provide a tool where incoming speech for the system is semi-automatically analyzed (human-in- the-loop) to identify sources of problems in the dialogue. The first step towards this goal was to build tools that auto- matically identify problematic dialogue situations or as we will call hereafter miscommunications. The automatic detection of miscommunications in SDSs has been extensively investigated in the literature (Walker et al., 2000; Swerts et al., 2000; Schmitt et al., 2010; Paek and Horvitz, 2004). This problem is vital in the develop- ment cycle of speech services. However, very little data is publicly available to perform research on this topic. One of the exceptions is (Swerts et al., 2000), but even in this case the dataset does not contain interactions with real users or annotations. In (Schmitt et al., 2012) a dataset collected with real users is described. This data was annotated for in- teraction quality (Schmitt et al., 2011), emotions and con- tains also a variety of automatically extracted features. This dataset was built with dialogues from CMU Let’s Go (Raux et …

Content truncated to 500 words