Open-Domain Sign Language Translation Learned from Online Video

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6365 - 6379 December 7-11, 2022 ©2022 Association for Computational Linguistics Open-Domain Sign Language Translation Learned from Online Video Bowen Shi TTI-Chicago bshi@ttic.edu Diane Brentari Univeristy of Chicago dbrentari@uchicago.edu Greg Shakhnarovich TTI-Chicago greg@ttic.edu Karen Livescu TTI-Chicago klivescu@ttic.edu Abstract Existing work on sign language translation— that is, translation from sign language videos into sentences in a written language—has fo- cused mainly on (1) data collected in a con- trolled environment or (2) data in a specific domain, which limits the applicability to real- world settings. In this paper, we introduce Ope- nASL, a large-scale American Sign Language (ASL) - English dataset collected from online video sites (e.g., YouTube). OpenASL contains 288 hours of ASL videos in multiple domains from over 200 signers and is the largest pub- licly available ASL translation dataset to date. To tackle the challenges of sign language trans- lation in realistic settings and without glosses, we propose a set of techniques including sign search as a pretext task for pre-training and fu- sion of mouthing and handshape features. The proposed techniques produce consistent and large improvements in translation quality, over baseline models based on prior work.1 1 Introduction Sign language, a type of visual language that con- veys meaning through gestures, is the most widely used form of linguistic communication among deaf and hard of hearing people. According to the World Health Organization, over 5% of the world’s population (∼430 million people) suffer from dis- abling hearing loss.2 Automatic sign language pro- cessing can facilitate the daily activities of deaf people and make artificial intelligence technolo- gies more accessible to deaf users. For example, such techniques would allow deaf users to interact with intelligent virtual assistants using sign lan- guage and would support automatic interpretation between sign languages and spoken languages. In- terest in sign language research has recently been 1Our data and code are publicly available at https:// github.com/chevalierNoir/OpenASL. 2https://www.who.int/news-room/fact-sheets/ detail/deafness-and-hearing-loss growing in the computer vision (CV) (Bragg et al., 2019; Adaloglou et al., 2021; Rastgoo et al., 2021) and natural language processing (NLP) communi- ties (Shterionov, 2021; Yin et al., 2021) In this paper, we study sign language transla- tion (SLT),3 the task of translating continuous sign- ing video into written language sentences. Un- like other sign language processing tasks such as sign spotting (Buehler et al., 2009) or continuous sign language recognition (sign-to-gloss transcrip- tion) (Dreuw et al., 2007), SLT has not been stud- ied until recently (Camgoz et al., 2018) and is still restricted to specific domains (e.g., weather forecasts (Camgoz et al., 2018), emergency situ- ations (Ko et al., 2019)), characterized by small vocabulary size and lack of visual variability. The lack of large-scale translation datasets in the wild is a central challenge in developing SLT technologies serving real-world use cases. In terms of translation modeling, most existing SLT approaches (Camgoz et al., 2018, 2020a,a; Zhou et al., 2021; Yin et al., 2021; Gan et al., 2021; Chen et …

Content truncated to 500 words