Deep JSLC: A Multimodal Corpus Collection for Data-driven Generation of Japanese Sign Language Expressions

Deep JSLC: A Multimodal Corpus Collection for Data-driven Generation of Japanese Sign Language Expressions Heike Brock, Kazuhiro Nakadai Honda Research Institute Japan Ltd. Honcho 8-1, Wako-shi, Saitama, Japan h.brock@jp.honda-ri.com, nakadai@jp.honda-ri.com Abstract The three-dimensional visualization of spoken or written information in Sign Language (SL) is considered a potential tool for better inclusion of deaf or hard of hearing individuals with low literacy skills. However, conventional technologies for such CG-supported data display are not able to depict all relevant features of a natural signing sequence such as facial expression, spatial references or inter-sign movement, leading to poor acceptance amongst speakers of sign language. The deployment of fully data-driven, deep sequence generation models that proved themselves powerful in speech and text applications might overcome this lack of naturalness. Therefore, we collected a corpus of continuous sentence utterances in Japanese Sign Language (JSL) applicable to the learning of deep neural network models. The presented corpus contains multimodal content information of high resolution motion capture data, video data and both visual and gloss-like mark up annotations obtained with the support of fluent JSL signers. Furthermore, all annotations were encoded under three different encoding schemes with respect to directions, intonation and non-manual information. Currently, the corpus is employed to learn first sequence-to-sequence networks where it shows the ability to train relevant language features. Keywords: sign language, deep learning corpus, assistive language technologies 1. Introduction Research has shown that information conveyed using Sign Language (SL) is much more comprehensible and acces- sible than written information display for the majority of native SL speakers (Traxler, 2000). However, human trans- lation services are neither always available on demand, nor applicable in certain settings such as internal company meetings. The automatic display of spoken or written con- tent by signing CG avatars or robot agents could there- fore offer a cheap possibility to make information more accessible to Deaf or Hard of Hearing (DHH) individuals with lower written language literacy (Huenerfauth, 2008). Unfortunately, automated SL generation technologies have not yet reached a level of quality that would foster their full acceptance by DHH users (Kipp et al., 2011b). This is mainly due to the language’s multi-dimensional char- acter: information is not only conveyed using single ges- tures and finger movements, but also Non-Manual Signs (NMSs) such as body language and facial expressions. Fur- thermore, lexical items within a signed conversation can be dynamically changed to express spatial and temporal infor- mation (Huenerfauth and Hanson, 2009). This creates non- linear, multichannel dependencies that (a) complicate the translation of SL as compared to spoken languages and (b) require specific descriptions for motion synthesis. As a re- sult, temporal and spatial relations of generated signing mo- tion sequences might easily appear artificial and unnatural when applied to three-dimensional avatar animations (Ka- corri et al., 2015). To increase the acceptance of three-dimensional sign an- imations within affected users, effort is made to develop new methodologies that can better represent the temporal, spatial and multimodal aspects of signed content informa- tion (Ebling and Huenerfauth, 2015; …

Content truncated to 500 words