Towards Understanding the Relation between Gestures and Language

Proceedings of the 29th International Conference on Computational Linguistics, pages 5507–5520 October 12–17, 2022. 5507 Towards Understanding the Relation between Gestures and Language Artem Abzaliev University of Michigan abzaliev@umich.edu Andrew Owens University of Michigan ahowens@umich.edu Rada Mihalcea University of Michigan mihalcea@umich.edu Abstract In this paper, we explore the relation between gestures and language. Using a multimodal dataset, consisting of TED talks where the lan- guage is aligned with the gestures made by the speakers, we adapt a semi-supervised mul- timodal model to learn gesture embeddings. We show that gestures are predictive of the native language of the speaker, and that ges- ture embeddings further improve language pre- diction result. In addition, gesture embed- dings might contain some linguistic informa- tion, as we show by probing embeddings for psycholinguistic categories. Finally, we ana- lyze the words that lead to the most expres- sive gestures and find that function words drive the expressiveness of gestures. Our code is available at https://github.com/ MichiganNLP/gestures-language. 1 Introduction Gestures are often referred to as “non-verbal lan- guage” and extensive studies in psychology, sociol- ogy, and anthropology have demonstrated the im- portant role they play in communication (McNeill, 1992; Iverson and Goldin-Meadow, 1998; Alibali et al., 2000). While language and gesture can occur independently, people often use them together to communicate, suggesting that multimodality plays an important role in understanding gestures. In this work, we consider human gestures together with their corresponding utterances. We jointly learn gesture and word embeddings, and attempt to pre- dict psycholinguistic categories and the language of the speaker from their gesture embeddings. Even for humans it is very challenging to pre- dict words from gestures alone (or vice versa), due to the many-to-many relationship between words and gestures. Therefore, instead of directly predict- ing one modality from the other (Desai and John- son, 2021), we use contrastive pre-training learning to learn a joint embedding space that aligns both modalities (Kiros et al., 2014; Tian et al., 2020; Radford et al., 2021). This allows our model to learn an association between language and gestures, despite a large amount of uncertainty inherent in the task. The main contributions of this work are as follows: • First, we explore a multimodal approach to learn gesture embeddings through contrastive learn- ing. Through validation experiments relying on these embeddings, we demonstrate that there is an association between gestures and languages representations. • Second, we probe gesture embeddings for vari- ous psychological and linguistic categories and show that gestures can be predictive of several categories with better-than-random accuracy. We find that function words, such as pronouns, prepo- sition or modal verbs, can be predicted from the gestures. We also show that gesture embeddings can be used to predict discourse markers. • Third, we show that it is possible to predict the language of a speaker from our learned gestures embeddings. Our findings indicate that the dif- ference in gestures across the languages may be driven by the function words. • Finally, we conduct several analyses to better understand …

Content truncated to 500 words