Prior Knowledge and Memory Enriched Transformer for Sign Language Translation

Findings of the Association for Computational Linguistics: ACL 2022, pages 3766 - 3775 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Prior Knowledge and Memory Enriched Transformer for Sign Language Translation Tao Jin Zhejiang University jint_zju@zju.edu.cn Zhou Zhao† Zhejiang University zhaozhou@zju.edu.cn Meng Zhang Huawei Noah’s Ark Lab zhangmeng92@huawei.com Xingshan Zeng Huawei Noah’s Ark Lab zxshamson@gmail.com Abstract This paper attacks the challenging problem of sign language translation (SLT), which in- volves not only visual and textual understand- ing but also additional prior knowledge learn- ing (i.e. performing style, syntax). How- ever, the majority of existing methods with vanilla encoder-decoder structures fail to suf- ficiently explore all of them. Based on this concern, we propose a novel method called Prior knowledge and memory Enriched Transformer (PET) for SLT, which incorpo- rates the auxiliary information into vanilla transformer. Concretely, we develop gated in- teractive multi-head attention which associates the multimodal representation and global sign- ing style with adaptive gated functions. One Part-of-Speech (POS) sequence generator re- lies on the associated information to predict the global syntactic structure, which is there- after leveraged to guide the sentence gener- ation. Besides, considering that the visual- textual context information, and additional auxiliary knowledge of a word may appear in more than one video, we design a multi-stream memory structure to obtain higher-quality translations, which stores the detailed corre- spondence between a word and its various rel- evant information, leading to a more compre- hensive understanding for each word. We con- duct extensive empirical studies on RWTH- PHOENIX-Weather-2014T dataset with both signer-dependent and signer-independent con- ditions. The quantitative and qualitative exper- imental results comprehensively reveal the ef- fectiveness of PET. 1 Introduction Recently, the combination of vision and language attracts increasing attention. Sign language transla- tion which aims to provide translated natural sen- tences for sign language videos is a valuable but challenging task in this topic (Camgoz et al., 2018, † corresponding author Translation: Im (ADP) | westen (NOUN) | ist (VERB) | es (PRON) | freundlich (ADJ) Figure 1: An example of sign language translation, where the video frames and the sentence correspond to each other. Besides, each word (red) has its syntactic attribute (green). 2020a,b; Jin and Zhao, 2021). Since the visual and textual modalities are not aligned strictly in a weakly-supervised manner, the difficulties of sign language translation mainly lie in the multimodal representation learning of both modalities and the alignments between them. Besides, additional prior knowledge (i.e. the performing style of different signers, the common syntactic structures of sen- tences) also has a strong influence on multimodal learning. Encoder-decoder structures built upon long short-term memory unit (Hochreiter and Schmidhu- ber, 1997) (LSTM) or transformer (Vaswani et al., 2017) are widely used in end-to-end sign language translation, which directly generates natural sen- tences without intermediate products like gloss se- quences. Generally, the encoder extracts and en- codes the sign language information, the decoder makes full use of the encoded results with cross- modal interaction. Camgoz et al. (2018) first pro- poses the sign …

Content truncated to 500 words