Gloss-Free End-to-End Sign Language Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12904–12916 July 9-14, 2023 ©2023 Association for Computational Linguistics Gloss-Free End-to-End Sign Language Translation Kezhou Lin1 Xiaohan Wang1 Linchao Zhu1 Ke Sun2 Bang Zhang2 Yi Yang1 1ReLER, CCAI, Zhejiang University 2DAMO Academy, Alibaba Group kezhoulin@zju.edu.cn wxh1996111@gmail.com zhulinchao7@gmail.com xisheng.sk@alibaba-inc.com bangzhang@gmail.com yangyics@zju.edu.cn Abstract In this paper, we tackle the problem of sign language translation (SLT) without gloss an- notations. Although intermediate representa- tion like gloss has been proven effective, gloss annotations are hard to acquire, especially in large quantities. This limits the domain cover- age of translation datasets, thus handicapping real-world applications. To mitigate this prob- lem, we design the Gloss-Free End-to-end sign language translation framework (GloFE). Our method improves the performance of SLT in the gloss-free setting by exploiting the shared un- derlying semantics of signs and the correspond- ing spoken translation. Common concepts are extracted from the text and used as a weak form of intermediate representation. The global em- bedding of these concepts is used as a query for cross-attention to find the corresponding information within the learned visual features. In a contrastive manner, we encourage the sim- ilarity of query results between samples con- taining such concepts and decrease those that do not. We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign.1 1 Introduction Sign language is a type of visual language mainly used by the community of deaf and hard of hearing. It uses a combination of hand gestures, facial ex- pressions, and body movements to convey the mes- sage of the signer. Sign languages are not simple transcripts of the corresponding spoken languages. They possess unique grammar structures and have their own linguistic properties. According to the World Federation of the Deaf, there are over 70 million deaf people around the world. The study of automated sign language processing can facilitate their day-to-day life. In this paper, we study the task of sign language translation (SLT), which translates the sign videos 1Our code and model will be available at https:// github.com/HenryLittle/GloFE. Figure 1: Use global embeddings of conceptual words (CA, conceptual anchor) in spoken translation to super- vise the visual feature instead of gloss. A and B are different samples from the same mini batch. into the corresponding spoken language. Glosses are the transliteration system for sign language. They serve as an intermediate representation of the signs. However, the vocabulary of gloss does not align with the spoken language nor does the order of the glosses. Unlike translation between two spoken languages, the number of frames in a sign video is much larger than the number of words in the spoken translation. This imposes a unique challenge for SLT. Models need to learn a clustering of the frames into gloss-level representa- tion before they can translate the tokens. Previous methods solve this problem in two major ways, i.e., pre-train the visual backbone with gloss (Cam- goz et al., 2020) or jointly train on both translation and continuous …

Content truncated to 500 words