Sign Language Translation: Ongoing Development, Challenges and Innovations in the SignON Project

Sign Language Translation: Ongoing Development, Challenges and Innovations in the SignON Project Dimitar Shterionov∗, Mirella De Sisto∗, Vincent Vandeghinste†,, Aoife Brady‡, Mathieu De Coster§, Lorraine Leeson¶, Josep Blat∗∗, Frankie Picron††, Marcello Paolo Scipioni‡‡, Aditya Parikh§§, Louis ten Bosch§§, John O’Flaherty∥, Joni Dambre§, Jorn Rijckaertx ∗Tilburg University, †Instituut voor de Nederlandse Taal, ‡ADAPT, §Ghent University ¶Trinity College Dublin,∗∗Universitat Pompeu Fabra,††European Union of the Deaf, ‡‡Fincons,§§Radboud University,∥mac.ie,xVlaams Gebarentaalcentrum 1 Introduction SignON1 focuses on the research and develop- ment of a sign language (SL) translation mobile application and an open communications frame- work. SignON addresses the lack of technol- ogy and services for the automatic translation between signed and spoken languages, through an inclusive, human-centric solution which facili- tates communication between deaf, hard of hearing (DHH) and hearing individuals. We present an overview of the status of the project, describing the milestones and the ap- proaches developed to address the challenges and peculiarities of SL machine translation (SLMT). SLs are the primary means of communication for over 70 million DHH individuals.2 Despite this, they are rarely included in ongoing develop- ments of natural-language processing (NLP) ad- vancements (Yin et al., 2021). Machine transla- tion (MT) research which targets SLs is still in its infancy, due mainly to the lack of data and effec- tive representation of signs (including the lack of a standardized written form for SLs). Both the low volume of available resources, as well as the linguistic properties of SLs provide challenges for MT. Furthermore, SLs are visual languages, which presents yet another challenge: © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1SignON is a Horizon 2020 (Research and Innovation Pro- gramme Grant Agreement No. 101017255) project that runs from 2021 until the end of 2023. https:// signon-project.eu/. The consortium is constituted by 17 partners, among which Instituut voor de Nederlandse Taal, Tilburg University, ADAPT, Ghent University, Trinity College Dublin, Universitat Pompeu Fabra, European Union of the Deaf, Fincons, Radboud University, mac.ie, Vlaams Gebarentaalcentrum, and Dublin City University. 2According to the World Federation of the Deaf. the recognition and synthesis of a signing human. 2 The SignON approach to SLMT The objective of the SignON project is MT be- tween signed and spoken languages in all possible combinations, as well as the delivery of this ser- vice to the primary user groups: DHH and hearing users. The project revolves around 4 spoken (English, Spanish, Dutch, Irish) and 5 SLs, (ISL, NGT, VGT, LSE, and BSL —namely Irish, Dutch, Flemish, Spanish and British SL). Addressing this many language pairs and directions on a pair-by-pair ba- sis would require a substantial amount of time and effort, far beyond the scope of the project. SignON employs an MT approach that (i) focuses on processing and understanding individual lan- guages, (ii) employs a common multi-lingual rep- resentation (InterL) to facilitate translation and (iii) uses symbolic as well as deep-learning methods for the synthesis of a 3D virtual signer. This ap- proach involves automatic …

Content truncated to 500 words