Neural Machine Translation Methods for Translating Text to Sign Language Glosses

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12523–12541 July 9-14, 2023 ©2023 Association for Computational Linguistics Neural Machine Translation Methods for Translating Text to Sign Language Glosses Dele Zhu1, Vera Czehmann1,2 and Eleftherios Avramidis2 1Technical University of Berlin, Berlin, Germany 2German Research Center for Artificial Intelligence (DFKI), Berlin, Germany dele.zhu@gmail.com, {vera.czehmann,eleftherios.avramidis}@dfki.de Abstract State-of-the-art techniques common to low re- source Machine Translation (MT) are applied to improve MT of spoken language text to Sign Language (SL) glosses. In our exper- iments, we improve the performance of the transformer-based models via (1) data augmen- tation, (2) semi-supervised Neural Machine Translation (NMT), (3) transfer learning and (4) multilingual NMT. The proposed methods are implemented progressively on two German SL corpora containing gloss annotations. Mul- tilingual NMT combined with data augmenta- tion appear to be the most successful setting, yielding statistically significant improvements as measured by three automatic metrics (up to over 6 points BLEU), and confirmed via human evaluation. Our best setting outperforms all pre- vious work that report on the same test-set and is also confirmed on a corpus of the American Sign Language (ASL). 1 Introduction Sign Language Translation (SLT) aims to break the language barrier between the deaf or hard-of- hearing communities and the hearing communi- ties. One challenging aspect of SLT is the fact that Sign Languages (SLs) are multi-channeled and non-written languages (Langer et al., 2014). There- fore, Machine Translation (MT) for SLs cannot directly take advantage of the recent developments in text-based MT. For this purpose, previous work has used written representations of the SLs. One of these representations are glosses, where signs are labeled by words of the corresponding spoken language, often including affixes and markers. It is known that glosses have strong limitations as a linguistic representation (Pizzuto et al., 2006). However, given the current status of SLT, we have indications that research on SL gloss translation can still be useful. For instance, translation from spoken language text to SL glosses can be use- ful for interpreters and educational uses (Collins Figure 1: Text-to-video SLT using glosses as an inter- mediate step (Source of images: Müller et al., 2020). et al., 2012). Secondly, SL glosses are the only SL representation having several parallel corpora big enough to train MT, and the results may provide indications for the future treatment of other more appropriate representations. Previous research on SLT has used glosses as an intermediate step to build MT systems for translating from SLs to spo- ken language text (Camgoz et al., 2017, 2018; Chen et al., 2023) or from spoken language text to SLs (Stoll et al., 2020; Saunders et al., 2020a,b, 2022). In the latter case, glosses allow building the system in two steps, i.e., text-to-gloss translation and gloss- to-video production (Figure 1). The glosses can be given to a system for the generation of SL (avatar animations, autoencoders, GANs). Our work fo- cuses on the first part of this pipeline, text-to-gloss translation, whose …

Content truncated to 500 words