Better Sign Language Translation with STMC-Transformer

Proceedings of the 28th International Conference on Computational Linguistics, pages 5975–5989 Barcelona, Spain (Online), December 8-13, 2020 5975 Better Sign Language Translation with STMC-Transformer Kayo Yin∗ Language Technologies Institute Carnegie Mellon University kayo@cmu.edu Jesse Read LIX, Ecole Polytechnique Institut Polytechnique de Paris jesse.read@polytechnique.edu Abstract Sign Language Translation (SLT) first uses a Sign Language Recognition (SLR) system to ex- tract sign language glosses from videos. Then, a translation system generates spoken language translations from the sign language glosses. This paper focuses on the translation system and introduces the STMC-Transformer which improves on the current state-of-the-art by over 5 and 7 BLEU respectively on gloss-to-text and video-to-text translation of the PHOENIX-Weather 2014T dataset. On the ASLG-PC12 corpus, we report an increase of over 16 BLEU. We also demonstrate the problem in current methods that rely on gloss supervision. The video- to-text translation of our STMC-Transformer outperforms translation of GT glosses. This contra- dicts previous claims that GT gloss translation acts as an upper bound for SLT performance and reveals that glosses are an inefficient representation of sign language. For future SLT research, we therefore suggest an end-to-end training of the recognition and translation models, or using a different sign language annotation scheme. 1 Introduction Communication holds a central position in our daily lives and social interactions. Yet, in a predominantly aural society, sign language users are often deprived of effective communication. Deaf people face daily issues of social isolation and miscommunication to this day (Souza et al., 2017). This paper is motivated to provide assistive technology that allow Deaf people to communicate in their own language. In general, sign languages developed independently of spoken language and do not share the grammar of their spoken counterparts (Stokoe, 1960). For this, Sign Language Recognition (SLR) systems on their own cannot capture the underlying grammar and complexities of sign language, and Sign Language Translation (SLT) faces the additional challenge of taking into account the unique linguistic features during translation. Figure 1: Sign language translation pipeline1. ∗*Work carried out while at ´Ecole Polytechnique. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 5976 As shown in Figure 1, current SLT approaches involve two steps. First, a tokenization system generates glosses from sign language videos. Then, a translation system translates the recognized glosses into spoken language. Recent work (Orbay and Akarun, 2020; Zhou et al., 2020) has addressed the first step, but there has been none improving the translation system. This paper aims to fill this research gap by leveraging recent success in Neural Machine Translation (NMT), namely Transformers. Another limit to current SLT models is that they use glosses as an intermediate representation of sign language. We show that having a perfect continuous SLR system will not necessarily improve SLT re- sults. We introduce the STMC-Transformer model performing video-to-text translation that surpasses translation of ground truth glosses, which reveals that glosses are a flawed representation of sign lan- guage. The contributions of this paper can be summarized as: 1. …

Content truncated to 500 words