The Automatic Annotation of the Semiotic Type of Hand Gestures in Obama’ s Humorous Speeches

The Automatic Annotation of the Semiotic Type of Hand Gestures in Obama’s Humorous Speeches Costanza Navarretta University of Copenhagen Emil Holms Kanal 2 2300 Copenhagen S - DK costanza@hum.ku.dk Abstract This paper describes a pilot study act to investigate the semiotic types of hand gestures in video-recorded speeches and their automatic classification. Gestures, which also comprise e.g. head movements and body posture, contribute to the successful delivery of the message by reinforcing what is expressed by speech or by adding new information to what is uttered. The automatic classification of the semiotic type of gestures from their shape description can contribute to their interpretation in human-human communication and in advanced multimodal interactive systems. We annotated and analysed hand gestures produced by Barack Obama during two speeches at the Annual White House Correspondent Dinners and found differences in the contexts in which various hand gesture types were used. Then, we trained machine learning algorithms to classify the semiotic type of the hand gestures. The F-score obtained by the best performing algorithm on the classification of four semiotic types is 0.59. Surprisingly, the shape feature that contributes mostly to classification is the trajectory of the left hand. The results of this study are promising, but they should be tested on more data of different type, produced by different speakers and in more languages. Keywords: Multimodal Communication, Automatic Classification, Hand Gestures 1. Introduction In face-to-face communication, people express their mes- sage both through the auditory modality, speech, and the vi- sual modality, gestures, which comprise head movements, body postures and hand gestures. Co-speech gestures are not redundant and contribute to the delivery of the mes- sage by reinforcing what it is said or providing new con- tent (McNeill, 2005; Kendon, 2004). Although the number of audio- and video-recorded speeches and conversations on the internet is growing every day, there is still a lack of freely available multimodally annotated data of differ- ent types of face-to-face communication. Since the manual annotation of gestures is extremely resource consuming, it is vital to address not only the automatic identification of occurrences of gestures and their physical characteristics, but also their interpretation. The automatic identification of hand gestural units from videos has been addressed by nu- merous studies, but identifying gestures from all kinds of videos is still not possible, and for this reason many studies use tracking devices or pose particular restrictions to light- ing and settings of the videos (Keskin et al., 2011). There- fore, there is still a need to identify the function of gestures automatically, and distinguishing their semiotic type is a first step toward their interpretation which should also in- clude the context in which the gestures are performed. This paper presents a pilot work aimed to the automatic classification of the semiotic type of hand and arm gestures in two speeches of Barack Obama using coarse-grained de- scriptions of the gestural shape. An analysis of hand ges- ture types in these data is also provided. The two humorous speeches were held …

Content truncated to 500 words