No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1884–1895 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1884 No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures Chaitanya Ahuja, Dong Won Lee, Ryo Ishii, Louis-Philippe Morency Language Technologies Institute Carnegie Mellon University Pittsburgh, PA {cahuja,dongwonl,rishii,lmorency}@cs.cmu.edu Abstract We study relationships between spoken lan- guage and co-speech gestures in context of two key challenges. First, distributions of text and gestures are inherently skewed mak- ing it important to model the long tail. Sec- ond, gesture predictions are made at a sub- word level, making it important to learn rela- tionships between language and acoustic cues. We introduce Adversarial Importance Sam- pled Learning (or AISLe), which combines ad- versarial learning with importance sampling to strike a balance between precision and cov- erage. We propose the use of a multimodal multiscale attention block to perform subword alignment without the need of explicit align- ment between language and acoustic cues. Fi- nally, to empirically study the importance of language in this task, we extend the dataset proposed in Ahuja et al. (2020) with auto- matically extracted transcripts for audio sig- nals. We substantiate the effectiveness of our approach through large-scale quantitative and user studies, which show that our proposed methodology significantly outperforms previ- ous state-of-the-art approaches for gesture gen- eration. Link to code, data and videos: https: //github.com/chahuja/aisle 1 Introduction Spoken language has gained more traction in the past decade due to improvements in natural lan- guage understanding and speech recognition. With an eye on the future, technologies such as intelli- gent personal assistants (e.g. Alexa, Siri, Cortana) are likely to also include embodiment to take advan- tage of the non-verbal communication that people naturally use in face-to-face interactions. As a step- ping stone in this direction, it is important to study the relationship between spoken language (which also includes acoustic information) and free form gestures (which go beyond just a pre-defined dic- tionary of gesture animations). In other words, how can we automatically generate human body pose (gestures) from language and acoustic inputs? An important technical challenge in such a natu- ral language processing task, is modeling the long tail of the language-gesture distribution (see Figure 1). If not addressed directly, computational models will likely focus on the common gestures (e.g beat gestures) as a way to improve precision at the cost of reduced coverage for less frequent words and gestures (Ginosar et al., 2019). Hence, when learn- ing these models, we need to not only be accurate for gesture generation, but also handle coverage of both linguistic and visual distributions (Pelachaud, 2009; Kucherenko et al., 2019). In other words, we need models that can balance precision and cover- age. Another technical challenge comes from the differences in granularity between language and gestures. Gestures can be triggered at the sub-word level; for example, by a change of intonation in acoustics. Thus, it is important to have sub-word level alignment between language and acoustics to generate the freeform …

Content truncated to 500 words