OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2114 - 2133 May 22-27, 2022 c⃝2022 Association for Computational Linguistics OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages Prem Selvaraj∗1, Gokul NC∗1,2, Pratyush Kumar1,2,3, Mitesh Khapra1,2 1AI4Bharat, 2IIT-Madras, 3Microsoft Research prem@ai4bharat.org, gokulnc@ai4bharat.org, pratyush@cse.iitm.ac.in, miteshk@cse.iitm.ac.in Abstract AI technologies for Natural Languages have made tremendous progress recently. However, commensurate progress has not been made on Sign Languages, in particular, in recognizing signs as individual words or as complete sen- tences. We introduce OpenHands1, a li- brary where we take four key ideas from the NLP community for low-resource languages and apply them to sign languages for word- level recognition. First, we propose using pose extracted through pretrained models as the stan- dard modality of data in this work to reduce training time and enable efficient inference, and we release standardized pose datasets for differ- ent existing sign language datasets. Second, we train and release checkpoints of 4 pose- based isolated sign language recognition mod- els across 6 languages (American, Argentinian, Chinese, Greek, Indian, and Turkish), provid- ing baselines and ready checkpoints for deploy- ment. Third, to address the lack of labelled data, we propose self-supervised pretraining on unla- belled data. We curate and release the largest pose-based pretraining dataset on Indian Sign Language (Indian-SL). Fourth, we compare dif- ferent pretraining strategies and for the first time establish that pretraining is effective for sign language recognition by demonstrating (a) improved fine-tuning performance especially in low-resource settings, and (b) high crosslin- gual transfer from Indian-SL to few other sign languages. We open-source all models and datasets in OpenHands with a hope that it makes research in sign languages reproducible and more accessible. 1 Introduction According to the World Federation of the Deaf, there are approximately 72 million Deaf people worldwide. More than 80% of them live in devel- oping countries. Collectively, they use more than ∗Equal contribution. 1https://github.com/AI4Bharat/ OpenHands 300 different sign languages varying across differ- ent nations (UN, 2021). Loss of hearing severely limits the ability of the Deaf to communicate and thereby adversely impacts their quality of life. In the current increasingly digital world, systems to ease digital communication between Deaf and hear- ing people are important accessibility aids. AI has a crucial role to play in enabling this accessibility with automated tools for Sign Language Recogni- tion (SLR). Specifically, transcription of sign lan- guage as complete sentences is referred to as Con- tinuous Sign Language Recognition (CSLR), while recognition of individual signs is referred to as Iso- lated Sign Language Recognition (ISLR). There have been various efforts to build datasets and mod- els for ISLR and CLSR tasks (Adaloglou et al., 2021; Koller, 2020). But these results are often concentrated on a few sign languages (such as the American Sign Language) and are reported across different research communities with few standard- ized baselines. When compared against text- and speech-based NLP research, the progress in AI re- search for sign …

Content truncated to 500 words