Improving American Sign Language Recognition with Synthetic Data

Improving American Sign Language Recognition with Synthetic Data Jungi Kim SYSTRAN Software, Inc. jungi.kim@systrangroup.com Patricia O’Neill-Brown U.S. Government po17b@icloud.com Abstract There is a need for real-time communica- tion between the deaf and hearing with- out the aid of an interpreter. Develop- ing a machine translation (MT) system be- tween sign and spoken languages is a mul- timodal task since sign language is a vi- sual language, which involves the auto- matic recognition and translation of video images. In this paper, we present the re- search we have been carrying out to build an automated sign language recognizer (ASLR), which is the core component of a machine translation (MT) system between American Sign Language (ASL) and En- glish. Developing an ASLR is a challeng- ing task due to the lack of sufficient quan- tities of annotated ASL-English parallel corpora for training, testing and develop- ing an ASLR. This paper describes the re- search we have been conducting to explore a range of different techniques for auto- matically generating synthetic data from existing datasets to improve the accuracy of ASLR. This work involved experimen- tation with several algorithms with varying amounts of synthetic data and evaluations of their effectiveness. It was demonstrated that automatically creating valid synthetic training data through simple image manip- ulation of ASL video recordings improves the performance of the ASLR task. c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. 1 Introduction In everyday life, there are situations in which there is the need for deaf and hearing individuals to com- municate with one another without the aid of an interpreter. To address this need, we are develop- ing ASL-English MT that enables signers and non- signers to communicate with one another using mobile devices such as smartphones and tablets. The concept is that the signer of ASL signs into the device and the video images are captured, auto- matically recognized, translated and rendered into both speech and text for the speaker of English. Conversely, using this application, the speaker’s speech is automatically recognized and an avatar signing the machine translation in ASL is dis- played, which appears along with the English text. This paper outlines our work on the first critical aspect of the problem, which is the development of an automatic sign language recognizer (ASLR). Specifically, we address our research in the area of generating valid synthetic data, a requirement dictated by the lack of sufficient amounts of large- scale annotated data for ASL to English for testing, training and developing ASLR algorithms. ASL is a visually perceived language based on a naturally evolved system of articulated hand ges- tures and their placement relative to the body, along with non-manual markers such as facial expressions, head movements, shoulder raises, mouth morphemes, and movements of the body (ASL: A brief description - Lifeprint.com). This language is structured like Japanese: it is a topic- comment language and does not have articles (Nakamura, 2008). See Speers (2002) for …

Content truncated to 500 words