Sign Language Recognition and Translation: A Multi-Modal Approach Using Computer Vision and Natural Language Processing

Proceedings of Recent Advances in Natural Language Processing, pages 658–665 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_071 658 Sign Language Recognition and Translation: A Multi-Modal Approach using Computer Vision and Natural Language Processing Jacky Li, Jaren Gerdes, James Gojit, Austin Tao, Samyak Katke, Kate Nguyen, and Benyamin Ahmadnia Department of Computer Engineering and Computer Science California State University, Long Beach, United States jacky.li01@student.csulb.edu, jaren.gerdes01@student.csulb.edu, james.gojit@student.csulb.edu, austin.tao01@student.csulb.edu, samyak.katke01@student.csulb.edu, kate.nguyen@student.csulb.edu, benyamin.ahmadnia@csulb.edu Abstract Sign-to-Text (S2T) is a hand gesture recogni- tion program in the American Sign Language (ASL) domain. The primary objective of S2T is to classify standard ASL alphabets and cus- tom signs and convert the classifications into a stream of text using neural networks. This paper addresses the shortcomings of pure Com- puter Vision techniques and applies Natural Language Processing (NLP) as an additional layer of complexity to increase S2T’s robust- ness. 1 Introduction Globally, sign language is one of the main lan- guages for those who cannot communicate verbally. Despite its global presence, not many people un- derstand it or use it. In 2020, 48 million people in the United States alone experience some form of hearing loss, with less than 500,000 – about 1% – of them that drive sign language regularly (Lacke, 2020; NIDCD, 2021). The World Health Organization (WHO) estimates that the number of individuals with hearing loss will affect nearly 2.5 billion by 2050 (WHO, 2023). With these setbacks, signers may find it challenging to communicate with other individuals not akin to their mode of communication. While mild hearing loss can be remedied with hearing aids and rehabilitation, these solutions may often be too expensive. Individuals can alterna- tively learn sign language. Hand gestures are a form of non-verbal communication used by indi- viduals in conjunction with speech to communicate. With the increasing use of technology, hand-gesture recognition is considered an essential aspect of Human-Machine Interaction (HMI), allowing the machine to capture and interpret the user’s intent and respond accordingly. The ability to discrimi- nate between human gestures can help in several applications that range from virtual and augmented reality to healthcare services (Ceolini et al., 2020). As technology becomes easier to use and ac- cessible, many people can likely perform simple commands with computer devices, such as typing text and video streaming. To address the problem statements, we propose S2T – a solution to close the sign language knowledge gap by translating simple hand gestures into text. 1.1 Sign-to-Text v1 The first Sign-to-Text (S2T) iteration was imple- mented using Computer Vision to classify the En- glish alphabet and custom gestures for text, such as space and delete. Computer Vision allows for gesture learning and recognition through images or video by identifying repeated patterns. Specific key descriptors can be isolated in a given frame using preprocessing techniques to eliminate noise and allow the neural network to perform on the highest data quality. While this process allows for the appropriate classification of newly introduced data, Computer Vision alone is not accurate enough to classify all ASL signs due to …

Content truncated to 500 words