Spatio-temporal Sign Language Representation and Translation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 977 - 982 December 7-8, 2022 ©2022 Association for Computational Linguistics DFKI-MLT at WMT-SLT22 Spatio-temporal Sign Language Representation and Translation Yasser Hamidullah and Josef van Genabith and Cristina España-Bonet {yasser.hamidullah,Josef.van_Genabith,cristinae}@dfki.de German Research Center for Artificial Intelligence (DFKI) Saarland Informatics Campus, Saarbrücken, Germany Abstract This paper describes the DFKI-MLT submis- sion to the WMT-SLT 2022 sign language trans- lation (SLT) task from Swiss German Sign Lan- guage (video) into German (text). State-of-the- art techniques for SLT use a generic seq2seq architecture with customized input embeddings. Instead of word embeddings as used in tex- tual machine translation, SLT systems use fea- tures extracted from video frames. Standard approaches often do not benefit from tempo- ral features. In our participation, we present a system that learns spatio-temporal feature rep- resentations and translation in a single model, resulting in a real end-to-end architecture ex- pected to better generalize to new data sets. Our best system achieved 5±1 BLEU points on the development set, but the performance on the test dropped to 0.11 ± 0.06 BLEU points. 1 Introduction Text-to-text machine translation (MT) is achiev- ing a great success with even (close to) human performance for some language pairs and do- mains (Akhbardeh et al., 2021). However, the sit- uation in sign language translation (SLT) is much different. One important reason is that the SLT is a low-resource scenario where one does not have the same amount of data as in high-resourced text- to-text to achieve a similar level of performance. A more specific reason is that SLT involves two modalities, text and video. Various problems arise when dealing with these modalities. Besides data scarcity, the lack of temporal boundaries in the in- put videos is a challenge. To overcome the lack of temporal boundaries, the most common solu- tion tends to ignore or not benefit from tempo- ral features. This approach relies on the Trans- former (Vaswani et al., 2017) capabilities to learn sequence-to-sequence tasks. The state-of-the-art SLT technique (Camgöz et al., 2020) is practically a normal Transformer but uses a custom embed- ding layer for 2D features extracted from video frames. In this approach, training a SLT system requires a pre-extraction step to convert frame features to vectors and train a Transformer sepa- rately to translate the vectors into spoken language. This type of approach has been widely used on a very specific dataset, the weather forecast corpus PHOENIX14T (Camgoz et al., 2018), where re- searchers reported a relatively good performance in terms of BLEU (∼20) (Camgöz et al., 2020; Min et al., 2021). Despite its good performance on a specific dataset, there is the doubt whether such type of architecture generalizes to new data sets. In or- der to build a more general technique, we focus on fundamental SLT problems such as the design, implementation and evaluation of a fully end-to- end model and representation learning for sign lan- guage videos. Having a fully end-to-end model fa- cilitates the task of data …

Content truncated to 500 words