A Study on the Ambiguity in Human Annotation of German Oral History Interviews for Perceived Emotion Recognition and Sentiment Analysis

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2022–2031 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2022 A Study on the Ambiguity in Human Annotation of German Oral History Interviews for Perceived Emotion Recognition and Sentiment Analysis Michael Gref 1, Nike Matthiesen2, Sreenivasa Hikkal Venugopala1,3, Shalaka Satheesh1,3 Aswinkumar Vijayananth1,3, Duc Bach Ha1, Sven Behnke1,4, Joachim K¨ohler1 1Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), Germany 2Haus der Geschichte der Bundesrepublik Deutschland Foundation (HdG), Bonn, Germany 3University of Applied Sciences Bonn-Rhein-Sieg, Germany 4Autonomous Intelligent Systems (AIS), Computer Science Institute VI, University of Bonn, Germany {michael.gref, sreenivasa.hikkal.venugopala, shalaka.satheesh, aswinkumar.vijayananth, duc.bach.ha, sven.behnke, joachim.koehler}@iais.fraunhofer.de, matthiesen@hdg.de Abstract For research in audiovisual interview archives often it is not only of interest what is said but also how. Sentiment analysis and emotion recognition can help capture, categorize and make these different facets searchable. In particular, for oral history archives, such indexing technologies can be of great interest. These technologies can help understand the role of emotions in historical remembering. However, humans often perceive sentiments and emotions ambiguously and subjectively. Moreover, oral history interviews have multi-layered levels of complex, sometimes contradictory, sometimes very subtle facets of emo- tions. Therefore, the question arises of the chance machines and humans have capturing and assigning these into predefined categories. This paper investigates the ambiguity in human perception of emotions and sentiment in German oral history interviews and the impact on machine learning systems. Our experiments reveal substantial differences in human perception for different emotions. Furthermore, we report from ongoing machine learning experiments with different modalities. We show that the human perceptual ambiguity and other challenges, such as class imbalance and lack of training data, currently limit the opportunities of these technologies for oral history archives. Nonetheless, our work uncovers promising observations and possibilities for further research. Keywords: emotion recognition, sentiment analysis, language, oral history, speech emotion recognition, facial emotion recognition, annotation, ambiguity 1. Introduction Oral history archives are often large audiovisual data repositories composing numerous interviews of con- temporary witnesses to historical events. Deep learn- ing can help to make these archives more accessible quantitatively and qualitatively. A prominent example in recent years is automatic speech recognition for tran- scription of oral history interviews. However, many potential deep learning applications for oral history interviews are still untapped. In a recent survey, Pessanha and Salah (2022) discuss potential ap- plications of computational technologies for oral his- tory archives. Among other aspects, the authors point out the potential benefits of these technologies in un- derstanding the changes in emotions during remem- bering, storytelling, and conversing. In conjunction with the transcriptions, researchers can better infer not only what is being said but also how. In a recent re- search project, we study sentiment analysis and emo- tion recognition for German oral history interviews as the foundation for such complex search and indexing approaches for archives. Various challenges arise when transferring research results to real-world, ”in-the-wild” applications. As with many AI-based approaches, suitable …

Content truncated to 500 words