Bidirectional Skeleton-Based Isolated Sign Recognition using Graph Convolutional Networks

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7328–7338 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7328 Bidirectional Skeleton-Based Isolated Sign Recognition using Graph Convolutional Networks Konstantinos M. Dafnis1, Evgenia Chroni1, Carol Neidle2, Dimitris N. Metaxas1 1 Rutgers University, 2 Boston University 1 110 Frelinghuysen Road, Piscataway, NJ 08854, 2 Boston University Linguistics, 621 Commonwealth Ave., Boston, MA 02215 kd703@cs.rutgers.edu, etc44@cs.rutgers.edu, carol@bu.edu, dnm@cs.rutgers.edu Abstract To improve computer-based recognition from video of isolated signs from American Sign Language (ASL), we propose a new skeleton-based method that involves explicit detection of the start and end frames of signs, trained on the ASLLVD dataset; it uses linguistically relevant parameters based on the skeleton input. Our method employs a bidirectional learning approach within a Graph Convolutional Network (GCN) framework. We apply this method to the WLASL dataset, but with corrections to the gloss labeling to ensure consistency in the labels assigned to different signs; it is important to have a 1-1 correspondence between signs and text-based gloss labels. We achieve a success rate of 77.43% for top-1 and 94.54% for top-5 using this modified WLASL dataset. Our method, which does not require multi-modal data input, outperforms other state-of-the-art approaches on the same modified WLASL dataset, demonstrating the importance of both attention to the start and end frames of signs and the use of bidirectional data streams in the GCNs for isolated sign recognition. Keywords: ASL, Isolated Sign Recognition, GCN, Linguistic Modeling 1. Introduction There are 28 million Deaf or Hard of Hearing peo- ple (Lin et al., 2011) in the US, and American Sign Language (ASL) is the primary language for ≥500,000 people (Mitchell et al., 2006). It is also the 3rd most studied “foreign” language (Looney and Lusin, 2019). Signed languages involve articulations of the hands and arms and non-manual expressions: facial expressions and movements of the head and upper body. Computer- based sign language recognition from video would pave the way for technologies to improve communi- cation between deaf and hearing individuals, such as ASL-to-English translation; educational applications to support ASL learners; or Google-like sign search by example over videos on the Web. These same technolo- gies could also be applied to other signed languages. The research reported here focuses on recognition of isolated, citation-form signs. The linguistically sig- nificant aspects of sign production include particu- lar hand configurations, palm orientations, locations (places of articulation), and movements, as well as non- manual components in some cases. As would be ex- pected in any language, the production of words (signs) shows considerable variability, and inter- and intra- signer variations pose a challenge for computer-based sign recognition. For this reason, it is important to use a large video corpus with sufficient numbers of exam- ples for each sign, including multiple signers. In this research, we used the WLASL dataset (Li et al., 2020), a collection of videos taken from various sources that includes 119 signers in 21,083 videos of 2,000 distinct isolated signs. …

Content truncated to 500 words