A Large Scale Speech Sentiment Corpus

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6549–6555 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6549 A Large Scale Speech Sentiment Corpus Eric Y. Chen1, Zhiyun Lu2∗, Hao Xu1, Liangliang Cao1, Yu Zhang1, James Fan1 1Google, Inc., New York, NY {erchen, hxx, llcao, ngyuzh, jjfan}@google.com 2University of Southern California Los Angeles, CA zhiyunlu@usc.edu Abstract We present a multimodal corpus for sentiment analysis based on the existing Switchboard-1 Telephone Speech Corpus released by the Linguistic Data Consortium. This corpus extends the Switchboard-1 Telephone Speech Corpus by adding sentiment labels from 3 different human annotators for every transcript segment. Each sentiment label can be one of three options: positive, negative, and neutral. Annotators are recruited using Google Cloud’s data labeling service and the labeling task was conducted over the internet. The corpus contains a total of 49500 labeled utterances covering 140 hours of audio. To the best of our knowledge, this is the largest multimodal Corpus for sentiment analysis that includes both speech and text features. Keywords: sentiment, switchboard, multimodal, speech 1. Introduction Sentiment analysis is the task of recognizing the sentiment of a given input. While most of the previous works done in sentiment analysis are using text based inputs, voice inputs are becoming more important as the adoption of voice based user agents, such as smart assistants and mo- bile voice control becomes more prevalent. In this paper we present the first large scale publicly available speech sentiment corpus, based on the Linguistic Data Consortium (LDC) Switchboard-1 Corpus. For the remainder of this paper, we shall refer to the original Switchboard corpus as Switchboard and our extension as Switchboard Sentiment. Each audio file in Switchboard is first split into segments based on the start and end time of the officially released transcript turns. These segments are henceforth referred to as utterances. We extend Switchboard by annotating each utterance with at least three human workers. Each worker may assign one of three sentiment labels to each utterance: positive, neutral, or negative. Workers are re- cruited over the internet using Google Cloud’s data label- ing service (Google, 2019). The data labeling task involves playing the audio corresponding to every utterance and ask workers to independently assign the most likely sentiment label. Every worker’s annotation is recorded and released with this corpus. Conflicting sentiment labels are a natural occurrence. We propose using a simple majority voting scheme to select the most probably sentiment label as the ground-truth. Based on this approach, the corpus has 30.4% positive utterances, 17% negative utterances, and 52.6% neutral utterances. Us- ing the highest voted sentiment label as ground-truth, we measure the average accuracy human workers at assigning the correct (highest voted) sentiment label as 85%. This number is lower than many of the other speech sentiment corpora. We suspect the reason is because most speech ∗The author performed the work while interned at Google. sentiment corpora either force participants to act out emo- tions following certain …

Content truncated to 500 words