Modeling Intensification for Sign Language Generation: A Computational Approach

Findings of the Association for Computational Linguistics: ACL 2022, pages 2897 - 2911 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Modeling Intensification for Sign Language Generation: A Computational Approach Mert ˙Inan1∗, Yang Zhong1*, Sabit Hassan1*, Lorna Quandt2, Malihe Alikhani1 1 Computer Science Department, School of Computing and Information, University of Pittsburgh, Pittsburgh, USA 2 Educational Neuroscience Program, Gallaudet University, Washington, D.C, USA {mert.inan, yaz118, sah259, malihe}@pitt.edu, lorna.quandt@gallaudet.edu Abstract End-to-end sign language generation models do not accurately represent the prosody in sign language. A lack of temporal and spatial varia- tions leads to poor-quality generated presenta- tions that confuse human interpreters. In this paper, we aim to improve the prosody in gen- erated sign languages by modeling intensifica- tion in a data-driven manner. We present dif- ferent strategies grounded in linguistics of sign language that inform how intensity modifiers can be represented in gloss annotations. To employ our strategies, we first annotate a sub- set of the benchmark PHOENIX-14T, a Ger- man Sign Language dataset, with different lev- els of intensification. We then use a super- vised intensity tagger to extend the annotated dataset and obtain labels for the remaining por- tion of it. This enhanced dataset is then used to train state-of-the-art transformer models for sign language generation. We find that our efforts in intensification modeling yield bet- ter results when evaluated with automatic met- rics. Human evaluation also indicates a higher preference of the videos generated using our model. 1 Introduction Similar to spoken languages, sign language has rich grammar rules and unique linguistic structures (Yin et al., 2021a; Emmorey, 2001). Elements of prosody such as rhythm, stress, or lengthening play important roles in distinguishing meaning and sig- naling intensification in sign language (Figure 1), similar to spoken languages (Brentari et al., 2018). Thus, it is important for sign language generation (SLG) systems to be able to learn accurately from the data and generate presentations that respect prosody. Much of the current study on prosodic markers such as intensifiers (Bolinger, 1972; Rett, 2008; ∗The first three authors have equal contribution. less clouds very cloudy WOLKE WOLKE 10 video frames 17 video frames Sign Not Repeated Sign Repeated No Delay Delayed Beginning Smaller Space Use Larger Space Use Figure 1: In sign language, modifiers are represented spatially and temporally. Here, two signers from PHOENIX-14T manually sign German "less clouds", and "very cloudy". Both of these signs have the same gloss representation: WOLKE (cloud in German). They are figuratively the same sign, but the duration, repeti- tion, temporal pauses, and continuations determine the exact meaning. This information is lost during sign lan- guage translation and evaluation. Ghesquière and Davidse, 2011) are based on lin- guistic theories of spoken languages and need to be adapted to signed languages, as prosody is repre- sented in the visual modality (Wennerstrom, 2001). Semantic differences are signaled in the visual modality using spatial and temporal presentations such as iconicity, gesture duration, as well as tem- poral pauses (Wilbur et al., 2012). Such distinctive properties present …

Content truncated to 500 words