Introducing the Weighted Trustability Evaluator for Crowdsourcing Exemplified by Speaker Likability Classification

Introducing the Weighted Trustability Evaluator for Crowdsourcing Exemplified by Speaker Likability Classification Simone Hantke1,2, Erik Marchi1,3, and Bj¨orn Schuller1,4 1 Chair of Complex & Intelligent Systems, University of Passau, Germany 2 Machine Intelligence & Signal Processing Group, Technische Universit¨at M¨unchen, Germany 3 audEERING UG, Gilching, Germany 4 Department of Computing, Imperial College London, UK Email: simone.hantke@uni-passau.de Abstract Crowdsourcing is an arising collaborative approach applicable among many other applications to the area of language and speech processing. In fact, the use of crowdsourcing was already applied in the field of speech processing with promising results. However, only few studies investigated the use of crowdsourcing in computational paralinguistics. In this contribution, we propose a novel evaluator for crowdsourced-based ratings termed Weighted Trustability Evaluator (WTE) which is computed from the rater-dependent consistency over the test questions. We further investigate the reliability of crowdsourced annotations as compared to the ones obtained with traditional labelling procedures, such as constrained listening experiments in laboratories or in controlled environments. This comparison includes an in-depth analysis of obtainable classification performances. The experiments were conducted on the Speaker Likability Database (SLD) already used in the INTERSPEECH Challenge 2012, and the results lend further weight to the assumption that crowdsourcing can be applied as a reliable annotation source for computational paralinguistics given a sufficient number of raters and suited measurements of their reliability. Keywords: Crowdsourcing, Speech Corpus Annotation, Computational Paralinguistics, Speech Classification 1. Introduction Computational paralinguistics deals with the computer- based analysis and synthesis of paralinguistic phenomena. As opposed to many related phenomena, such as speakers states and traits, the term involves related fields such as so- cial signal processing (Vinciarelli et al., 2012) and affec- tive computing (Picard, 2000). A suitable definition can be potentially given in an ‘ex-negativo’ fashion: it comprises everything that is not dealt within phonetics or linguistics (Schuller and Batliner, 2014). The success of supervised machine learning techniques for paralinguistic tasks depends highly on the quality of the ex- isting labelled training data and therefore on the quality of the labels. The manual annotation of data by an expert is the primary way of gathering labelled training data for speech recognition. This way of getting the labels can be very time-consuming and expensive (Tarasov et al., 2010; Ambati et al., 2010; Hsueh et al., 2009; Kittur et al., 2008). Furthermore, the labels have to be estimated from the sub- jective opinion of a small number of experts who can often disagree on the labels (Tarasov et al., 2010; Donmez et al., 2009; Raykar et al., 2010). This paper proposes an alternative way to gather anno- tated data for paralinguistic tasks with the use of non- professional annotators via crowdsourcing. Recently, with the access to crowdsourcing services such as Mechanical Turk1 and CrowdFlower2, it has become easier to get labels from multiple non-expert annotators. Laboratory studies allow us, as researchers, to control many variables during the experiments. However, there are con- siderable differences in the environment between crowd- 1https://www.mturk.com 2http://www.crowdflower.com sourcing workers and the traditional laboratory …

Content truncated to 500 words