Sign Language Production With Avatar Layering: A Critical Use Case over Rare Words

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1519–1528 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1519 Sign Language Production With Avatar Layering: A Critical Use Case over Rare Words Jung-Ho Kim1 Eui Jun Hwang1 Sukmin Cho1 Du Hui Lee2 Jong C. Park1 1KAIST Daehak-ro 291, Yuseong-gu, Daejeon, Republic of Korea {jhkim, ejhwang, nelllpic, park}@nlp.kaist.ac.kr 2EQ4ALL Nonhyeon-ro 76-gil 11, Gangnam-gu, Seoul, Republic of Korea scottlee@eq4all.co.kr Abstract Sign language production (SLP) is the process of generating sign language videos from spoken language expressions. Since sign languages are highly under-resourced, existing vision-based SLP approaches suffer from out-of-vocabulary (OOV) and test-time generalization problems and thus generate low-quality translations. To address these problems, we introduce an avatar-based SLP system composed of a sign language translation (SLT) model and an avatar animation generation module. Our Transformer-based SLT model utilizes two additional strategies to resolve these problems: named entity transformation to reduce OOV tokens and context vector generation using a pretrained language model (e.g., BERT) to reliably train the decoder. Our system is validated on a new Korean-Korean Sign Language (KSL) dataset of weather forecasts and emergency announcements. Our SLT model achieves an 8.77 higher BLEU-4 score and a 4.57 higher ROUGE-L score over those of our baseline model. In a user evaluation, 93.48% of named entities were successfully identified by participants, demonstrating marked improvement on OOV issues. Keywords: sign language, sign language production, signing avatar, Korean, Korean Sign Language 1. Introduction Sign language (SL) is the primary communication method for the Deaf1 community. Unlike spoken lan- guage, sign language conveys meaning via the move- ments of hands, face, and body. Moreover, sign lan- guage has a different grammar and lexicon from the lo- cal spoken language (Sandler and Lillo-Martin, 2006). Therefore, Deaf people usually need a sign language interpreter to interact in hearing society. However, the total need for interpreting services exceeds service availability, especially for translation contexts consid- ered low-priority. Furthermore, access to interpreting services is crucial during disaster and emergency sit- uations where interpreters cannot be requested in ad- vance. As an alternative, signing avatars have been widely researched for delivering information in sign lan- guage (Huenerfauth, 2008; Ebling and Huenerfauth, 2015; Kacorri and Huenerfauth, 2016). Avatar-based animation offers flexibility when creating new expres- sions but are not well accepted by the Deaf commu- nity due to the lack of fluency and naturalness in gen- erated animations. Recently, end-to-end vision-based sign language production (SLP) has received a lot of attention and has shown significant progress (Stoll et al., 2020; Saunders et al., 2021; Hwang et al., 2021). However, it is still challenging to generalize these mod- 1The uppercase “Deaf” refers to those deaf people who share a sign language and a culture, and the lowercase “deaf” refers to the condition of not hearing (Padden and Humphries, 1989). Named Entity Transformation 오늘 16시 서울지역 대설주의보 발효예정입니다. (Heavy snow advisory will take effect in Seoul today at 16:00.) 오늘 __NUM시 __NE지역 대설·강풍주의보 발효예정입니다. (Heavy …

Content truncated to 500 words