Clean Text and Full-Body Transformer: Microsoft’s Submission to the WMT22 Shared Task on Sign Language Translation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 969 - 976 December 7-8, 2022 ©2022 Association for Computational Linguistics Clean Text and Full-Body Transformer: Microsoft’s Submission to the WMT22 Shared Task on Sign Language Translation *Subhadeep Dey, Abhilash Pal, Cyrine Chaabani, *Oscar Koller Microsoft - Munich, Germany {subde,t-apal,t-cchaabani,oskoller}@microsoft.com Abstract This paper describes Microsoft’s submission to the first shared task on sign language translation at WMT 2022, a public competition tackling sign language to spoken language translation for Swiss German sign language. The task is very challenging due to data scarcity and an un- precedented vocabulary size of more than 20k words on the target side. Moreover, the data is taken from real broadcast news, includes native signing and covers scenarios of long videos. Motivated by recent advances in action recog- nition, we incorporate full body information by extracting features from a pre-trained I3D model and applying a standard transformer net- work. The accuracy of the system is further improved by applying careful data cleaning on the target text. We obtain BLEU scores of 0.6 and 0.78 on the test and dev set respectively, which is the best score among the participants of the shared task. Also in the human evalua- tion the submission reaches the first place. The BLEU score is further improved to 1.08 on the dev set by applying features extracted from a lip reading model. 1 Introduction Sign languages are natural visual languages that are used by deaf and hard of hearing individuals to communicate in everyday life. Sign languages are actively being researched. However, there is a huge imbalance in the field of natural language and speech processing between oral and signed languages. Since recently, one observes the emer- gence of a transition shifting sign language pro- cessing to be part of the NLP mainstream (Yin et al., 2021). We embrace this development which manifests (among others) in the creation of the first shared task on sign language translation as part of WMT 2022 (Mathias et al., 2022). It is great to have real-world sign language data (Bragg et al., 2019; Yin et al., 2021) as the basis of this *Equal Contribution shared task, manifested in native signers content and an unprecedentedly large vocabulary. Never- theless, this leads to a very challenging task with low performance numbers. When participating in the advances of sign language technologies it is worth recapping that deaf people have much at stake, both to gain and lose from applications that will be enabled here (Bragg et al., 2021). We aim to advance the field and the use-cases in a positive way and present our findings in this system paper. In the remainder of this work we first present a brief view on the relevant literature in Section 2, then we present the employed data in Section 3. Subsequently, we describe our submission in Sec- tion 4, additional experiments in Section 5 and we end with a summary in Section 6. 2 Related Work In this section, we present a …

Content truncated to 500 words