SignON: Sign Language Translation. Progress and challenges.

SignON Sign Language Translation: Progress and Challenges Vincent Vandeghinste†a, Dimitar Shterionov∗, Mirella De Sisto∗, Aoife Brady‡, Mathieu De Coster§ Lorraine Leeson¶, Josep Blat∗∗, Frankie Picron††, Marcello Paolo Scipioni‡‡ Aditya Parikh§§, Louis ten Bosch§§, John O’Flaherty∥, Joni Dambre§, Jorn Rijckaertx Bram Vanroya, Victor Ubieto Nogales∗∗, Santiago Egea Gomez∗∗, Ineke Schuurmana Gorka Labakab, Adri´an N´u˜nez-Marcosb, Irene Murtaghc,Euan McGill∗∗, Horacio Saggion∗∗ ∗Tilburg University, †Instituut voor de Nederlandse Taal, ‡ADAPT, §Ghent University, ¶Trinity College Dublin,∗∗Universitat Pompeu Fabra,††European Union of the Deaf, ‡‡Fincons,§§Radboud University,∥mac.ie,xVlaams Gebarentaalcentrum, aKU Leuven, bUniversity of the Basque Country UPV/EHU, cTU Dublin SignON1 is a Horizon 20202 project, running from January 2021 until December 2023, address- ing the lack of technology and services for MT be- tween sign languages (SLs) and spoken languages (SpLs), through an inclusive, human-centric solu- tion, contributing to the repertoire of communica- tion media for deaf, hard of hearing (DHH) and hearing individuals. Even though there are esti- mates that over 70 million DHH individuals have SLs as their primary means of communication, SLs are often not targeted by new language tech- nologies, due to challenges, such as the scarcity of data and the lack of a standardized written rep- resentation. This paper presents an update of the project status, describing how we address the chal- lenges and peculiarities of SLMT. We built an MT framework between SLs and SpLs, in all possible combinations, focusing on Irish, Dutch, Flemish, Spanish and British SL and on Irish, Dutch, Spanish and English SpLs (spoken and written). To limit the computational complex- ity and allow the effective development of compo- nents in parallel, we develop a translation pipeline that employs an interlingual representation (In- terL) (Figure 1). Inputs can be an SpL utterance in audio or text or an SL utterance in video. The input is processed via the corresponding compo- nent: automatic speech recognition (ASR) con- verts audio into text; SL recognition (SLR) con- verts SL videos into latent representations. The in- tegration of all of these components is currently ongoing. We develop ASR for both typical and atypical speech, such as speech of DHH persons. © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. 1https://signon-project.eu/ 2Research and Innovation Programme Grant Agreement No. 101017255 A use case sub-project collects speech data from this specific user group. Both conventional ‘mod- ular’ approaches as well as more recently devel- oped end-to-end approaches based on deep learn- ing (DL) are employed. SLR uses a pose estimator (Lugaresi et al., 2019) and post-processing of the predicted key- points. This yields robust representations: miss- ing data are imputed and keypoints are normalised to account for camera position. These representa- tions are further processed into embeddings, which are fine-tuned on SL data, using glosses as target labels. However, we do not predict glosses but ex- tract visual embeddings which are used as input for the SL MT models. We use mBART (Liu et al., 2020) for text-to- text translation, fine-tuned to also support Irish and …

Content truncated to 500 words