Creating a Corpus of Gestures and Predicting the Audience Response based on Gestures in Speeches of Donald Trump

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1081–1088 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1081 Creating a Corpus of Gestures and Predicting the Audience Response based on Gestures in Speeches of Donald Trump Verena Ruf1,2, Costanza Navarretta1 1University of Copenhagen, 2 Technical University Kaiserslautern vruf@physik.uni-kl.de, costanza@hum.ku.dk Abstract Gestures are an important component of non–verbal communication. This has an increasing potential in human–computer interaction. For example, Navarretta (2017b) uses sequences of speech and pauses together with co–speech gestures produced by Barack Obama in order to predict audience response, such as applause. The aim of this study is to explore the role of speech pauses and gestures alone as predictors of audience reaction without other types of speech information. For this work, we created a corpus of speeches held by Donald Trump before and during his time as president between 2016 and 2019. The data were transcribed with pause information and co–speech gestures were annotated as well as audience responses. Gestures and long silent pauses of the duration of at least 0.5 seconds are the input of computational models to predict audience reaction. The results of this study indicate that especially head movements and facial expressions play an important role and they confirm that gestures can to some extent be used to predict audience reaction independently of speech. Keywords: Multimodal Communication, Machine Learning, Audience Response 1. Introduction Conscious or subconscious gesturing is part of non–verbal communication (Argyle, 2010). Therefore, gestures, such as hand gestures, head movements, facial expressions or body posture, which are connected to speech (Kendon, 2004; McNeill, 2015), are called co–speech gestures. Politicians also use gestures and, because they frequently hold public speeches about their goals and plans, their speeches are often available online and have been objects of various studies. Navarretta (2017b) analyses two speeches by Barack Obama at the White House Correspondents’ As- sociation Dinner and finds that sequences of speech, pauses and these co–occurring gestures can be employed to predict audience response using machine learning. The content of the spoken sequences is not included in her study. The aim of this paper is to investigate whether long speech pauses and co–speech gestures alone contribute to the prediction of audience responses in speeches by a speaker different from Obama. Accordingly, the focus of this work lies on co–speech gestures and pauses. The findings could, for in- stance, be applied in future research concerning multimodal communication with robots or other communicative inter- faces regarding, for example, the extent of gesturing neces- sary for eliciting a response from an interlocutor. The paper is organised as follows: First, in section 2., back- ground literature is discussed. Section 3. contains a de- scription of the data, section 4. includes a qualitative anal- ysis of a short extract and section 5. presents the computa- tional models used in the prediction experiments. A discus- sion completes the paper (section 6.). 2. Background The two main purposes of political speeches are to …

Content truncated to 500 words