Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 994 - 1000 December 7-8, 2022 ©2022 Association for Computational Linguistics Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22 Laia Tarrés1,2∗and Gerard I. Gállego1,∗ and Xavier Giró-i-Nieto1 and Jordi Torres2 1Universitat Politècnica de Catalunya, Spain 2Barcelona Supercomputing Center, Spain Abstract This paper describes the system developed at the Universitat Politècnica de Catalunya for the Workshop on Machine Translation 2022 Sign Language Translation Task, in particular, for the sign-to-text direction. We use a Trans- former model implemented with the Fairseq modeling toolkit. We have experimented with the vocabulary size, data augmentation tech- niques and pretraining the model with the PHOENIX-14T dataset. Our system obtains 0.50 BLEU score for the test set, improving the organizers’ baseline by 0.38 BLEU. We re- mark the poor results for both the baseline and our system, and thus, the unreliability of our findings. 1 Introduction The submission of the Universitat Politècnica de Catalunya (UPC) to the WMT22 Sign Lan- guage Translation (SLT) Task experimented with vocabulary size, data augmentation techniques and a pretrained system with the PHOENIX-14T dataset (Camgoz et al., 2018). Up to the author’s knowledge, our implementation is the first to build on Fairseq, a popular modeling toolkit by Meta AI (Ott et al., 2019). SLT is a highly complex task because sign lan- guage understanding requires a very precise esti- mation of the signer pose, especially, of its hands. In addition, sign languages have grammatical struc- tures different from spoken languages, which pre- vents an easy knowledge transfer. Sign languages are represented in continuous and high-dimensional spaces, while the transcribed version of spoken lan- guages are represented by discrete tokens of well- defined vocabularies. Moreover, the few available sign language datasets can be considered very low- resourced, as they typically contain less than one hundred thousand sentences (Goyal et al., 2022). ∗Equal contribution In particular, the total number of sentences from the two datasets provided in WMT-SLT22 is 17k. We focus on the sign language translation task of WMT-SLT22 which requires participants to pre- dict the translation in spoken language (written) from a sign language video. Specifically, it consists of translation from Swiss German Sign Language (DSGS) videos to German (DE) text. The organizers of WMT 2022 SLT track pro- pose a Transformer baseline (Müller et al., 2022) that achieves a very low BLEU (Papineni et al., 2002) score,1 which indicates a very poor transla- tion quality. The training data provided consists of two datasets: FocusNews (Müller et al., 2022b) and SRF (Müller et al., 2022a). The first one contains 197 episodes in DSGS that have an average length of 5 minutes, which amount for a total duration of 19 hours. The second dataset contains 29 videos from live sign language interpretation, that have an average of 30 minutes length, and totals a duration of 16 hours. The paired subtitles are given in Stan- dard German from Switzerland, which is a dialect of German. The organizers provide keypoints extracted with two off-the-shelf …

Content truncated to 500 words