TTIC’s WMT-SLT 22 Sign Language Translation System

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 989 - 993 December 7-8, 2022 ©2022 Association for Computational Linguistics TTIC’s WMT-SLT 22 Sign Language Translation System Bowen Shi TTI-Chicago bshi@ttic.edu Diane Brentari Univeristy of Chicago dbrentari@uchicago.edu Greg Shakhnarovich TTI-Chicago greg@ttic.edu Karen Livescu TTI-Chicago klivescu@ttic.edu Abstract We describe TTIC’s model submission to WMT-SLT 2022 task (Müller et al., 2022) on sign language translation (Swiss-German Sign Language (DSGS) → German). Our model consists of an I3D backbone for image encod- ing and a Transformer-based encoder-decoder model for sequence modeling. The I3D is pre- trained with isolated sign recognition using the WLASL dataset. The model is based on RGB images alone and does not rely on the pre-extracted human pose. We explore a few different strategies for model training in this paper. Our system achieves 0.3 BLEU score and 0.195 Chrf score on the official test set. 1 Introduction Sign language, a full-fledged natural language that conveys meaning through gestures, is the primary chief of communication among Deaf people. Sign language translation is a task for automatically translating sign languages into written languages. Due to its widespread potential applications, it has recently received growing research interest (Cam- goz et al., 2018, 2021). Existing methods for sign language translation are primarily based on gloss, a transliteration sys- tem annotating sign language with symbols from written language. Utilizing gloss usually boosts the performance of current translation systems by a large margin. In the widely used German sign language translation benchmark Phoenix14T (Cam- goz et al., 2018), state-of-the-art gloss-based mod- els (Chen et al., 2022) are roughly 15 points better (in Bleu-4) than gloss-free models (Camgoz et al., 2018). However, gloss is more expensive to anno- tate than written language translation. There have been relatively few amounts of studies for gloss- free sign language translation. Specifically, Orbay and Akarun (2020); Shi et al. (2022) utilize local visual features (e.g., hands) to enhance the transla- tion performance. Those systems require domain- specific training data (e.g., labeled handshape data used in Orbay and Akarun (2020)), which is not always accessible for the target sign language. The fusion of visual features at different scales also increases the complexity of the modeling pipeline. In this paper, we study a simple model for sign language translation between DSGS and German in a gloss-free setting. Our model uses a 3D convo- lutional network for visual feature extraction and a Transformer-based encoder-decoder for sequence modeling. It is built on raw RGB images rather than pose keypoints, thus avoiding potential mis- takes from pose estimation and remaining fast in inference. We further study the impact of hyper- parameters and different pretraining strategies on translation quality. Without ensembling, our model achieves 0.3 Bleu score and 0.195 Chrf score on the official test set. 2 Method In this section, we describe our method for sign language translation. Our model consists of an In- flated 3D ConvNet (I3D) (Carreira and Zisserman, 2017) for visual encoding and a Transformer-based encoder-decoder model (Vaswani et al., 2017) for …

Content truncated to 500 words