Experimental Machine Translation of the Swiss German Sign Language via 3D Augmentation of Body Keypoints

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 983 - 988 December 7-8, 2022 ©2022 Association for Computational Linguistics Experimental Machine Translation of the Swiss German Sign Language via 3D augmentation of body keypoints Lorenz Hufe and Eleftherios Avramidis German Research Center for Artificial Intelligence (DFKI Berlin) Speech and Language Technology, Alt Moabit 91c, 10559 Berlin lorenz.hufe@web.de, eleftherios.avramidis@dfki.de Abstract This paper describes the participation of DFKI- SLT at the Sign Language Translation Task of the Seventh Conference of Machine Translation (WMT22). The system focuses on the trans- lation direction from the Swiss German Sign Language (DSGS) to written German. The orig- inal videos of the sign language were analyzed with computer vision models to provide 3D body keypoints. A deep-learning sequence-to- sequence model is trained on a parallel cor- pus of these body keypoints aligned to writ- ten German sentences. Geometric data aug- mentation occurs during the training process. The body keypoints are augmented by artificial rotation in the three dimensional space. The 3D-transformation is calculated with different angles on every batch of the training process. 1 Introduction Despite the enormous progress of the Machine Translation (MT) of spoken (and written) lan- guages, the MT of sign languages is in a very early stage (Yin et al., 2021; De Coster et al., 2022). Two major challenges are (a) the multimodal and mul- tilateral nature of the sign languages and (b) the lack of data. On the one side, the multilateral and multimodal nature of the sign languages requires deep-learning topologies that differ substantially from the ones used in text-based MT. On the other side, the lack of data makes difficult the utiliza- tion of end-to-end deep learning algorithms, which usually require vast amounts of data. As a result, deep-learning experiments have been executed for very few sign languages (e.g. German Sign Lan- guage, DGS; American Sign Language, ASL) and narrow domains (e.g. weather forecasts), leaving open questions on the generalization of the meth- ods to other sign languages and broader domains. This year’s Sign Language Translation (SLT) Task of the Seventh Conference of Machine Trans- lation (WMT22) is contributing significant to this direction, by adding a new language pair (Swiss German Sign Language - DGSG - to German) and allowing extensive experimentation from several participants on the same dataset. Our system uses computer vision models to ana- lyze the sign language videos into body keypoints and uses these keypoints as the source-side input of the neural MT transformer, allowing to perform data augmentation via geometrical augmentations. Despite the difficulty of this shared task and the low results obtained, we publish this paper as a techni- cal report, with the hope that it can contribute to the further research of this direction. The rest of the paper is organized as following. Section 2 positions our contribution amidst related work. Section 3 describes the methods for training the system and Section 4 the technical set-up of the experiment. Section 5 provides and discusses some results, while Section 6 gives some …

Content truncated to 500 words