Challenges with Sign Language Datasets for Sign Language Recognition and Translation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2478–2487 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2478 Challenges with Sign Language Datasets for Sign Language Recognition and Translation Mirella De Sisto∗, Vincent Vandeghinste†, Santiago Egea G´omez‡ Mathieu De Coster§, Dimitar Shterionov∗, Horacio Saggion‡ ∗Tilburg University, the Netherlands m.desisto, d.shterionov@tilburguniversity.edu †Dutch Language Institute, Leiden, the Netherlands Centre for Computational Linguistics, Leuven.AI, KU Leuven, Belgium vincent.vandeghinste@ivdnt.org ‡Universitat Pompeu Fabra, Barcelona, Spain santiago.egea, horacio.saggion@upf.edu §IDLab-AIRO – Ghent University – imec, Ghent, Belgium mathieu.decoster@ugent.be Abstract Sign Languages (SLs) are the primary means of communication for at least half a million people in Europe alone. However, the development of SL recognition and translation tools is slowed down by a series of obstacles concerning resource scarcity and standardization issues in the available data. The former challenge relates to the volume of data available for machine learning as well as the time required to collect and process new data. The latter obstacle is linked to the variety of the data, i.e., annotation formats are not unified and vary amongst different resources. The available data formats are often not suitable for machine learning, obstructing the provision of automatic tools based on neural models. In the present paper, we give an overview of these challenges by comparing various SL corpora and SL machine learning datasets. Furthermore, we propose a framework to address the lack of standardization at format level, unify the available resources and facilitate SL research for different languages. Our framework takes ELAN files as inputs and returns textual and visual data ready to train SL recognition and translation models. We present a proof of concept, training neural translation models on the data produced by the proposed framework. Keywords: sign language translation, sign language recognition, sign language corpora, unified data format, machine learn- ing 1. Introduction More than 70 million people worldwide are deaf.1 In Europe alone, for approximately half a million of deaf and hard of hearing (DHH) people, Sign Languages (SLs) are the main or preferred means of communi- cation (Pasikowska-Schnass, 2018). SLs are natural languages which exploit the visual-gestural channel. Ethnologue lists 150 sign languages (Eberhard et al., 2021); however, this list is far from exhaustive, and many SLs remain undocumented. Even in the case of recognised and described SLs, the limited amount of data available relegates them into the category of low resource languages. In recent years, there has been a growing interest in ap- plying natural language processing techniques to SLs (Camgoz et al., 2018; Camgoz et al., 2020; De Coster et al., 2021; Zhou et al., 2021; Yin et al., 2021; Moryossef et al., 2021) and, in particular, in devel- oping tools for automatic translation between SLs and spoken languages. Namely, the progress in deep learn- 1Estimation from the World Federation of the Deaf (WFD) https://wfdeaf.org/ ing has drawn an inspiring context to develop tools to automatically translate SLs. The two main tasks in- volved to automatically translate signs to spoken …

Content truncated to 500 words