Scaling Back-Translation with Domain Text Generation for Sign Language Gloss Translation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 463–476 May 2-6, 2023 ©2023 Association for Computational Linguistics Scaling Back-Translation with Domain Text Generation for Sign Language Gloss Translation Jinhui Ye1∗ Wenxiang Jiao2∗ Xing Wang2† Zhaopeng Tu2 1The Hong Kong University of Science and Technology (Guangzhou) jye624@connect.hkust-gz.edu.cn 2Tencent AI Lab {joelwxjiao,brightxwang,zptu}@tencent.com Abstract Sign language gloss translation aims to trans- late the sign glosses into spoken language texts, which is challenging due to the scarcity of la- beled gloss-text parallel data. Back-translation (BT), which generates pseudo parallel data by translating in-domain spoken language texts into sign glosses, has been applied to alleviate the data scarcity problem. However, the lack of large-scale high-quality in-domain spoken language text data limits the effect of BT. In this paper, to overcome the limitation, we pro- pose a Prompt based domain text Generation (PGEN) approach to produce the large-scale in-domain spoken language text data. Specifi- cally, PGEN randomly concatenates sentences from the original in-domain spoken language text data as prompts to induce a pre-trained lan- guage model (i.e., GPT-2) to generate spoken language texts in similar style. Experimental results on three benchmarks of sign language gloss translation in varied languages demon- strate that BT with spoken language texts gen- erated by PGEN significantly outperforms the compared methods. In addition, as the scale of spoken language texts generated by PGEN increases, the BT technique can achieve further improvements, demonstrating the effectiveness of our approach. We release the code and data for facilitating future research in this field1. 1 Introduction Sign language is the dominant form of communica- tion for the deaf and hearing impaired community. Sign language processing has received substantial attention in the last few years and achieved signifi- cant progress (Bragg et al., 2019; Yin et al., 2021; Shterionov, 2021; De Coster et al., 2022). Among ∗Jinhui Ye and Wenxiang Jiao contributed equally to this work. Work was mainly done when Jinhui Ye was interning at Tencent AI Lab. †Xing Wang is the corresponding author. 1Code and data are available at https://github.com/ Atrewin/PGen. Valid BLEU vs Scale BLEU 19 21 23 25 27 Scaling Times 0 11 22 33 44 55 T-sign T-gpt T-retrieval T-general Val BLEU 23.0 23.5 24.0 24.5 25.0 0 5 10 Valid BLEU vs Scale BLEU 19 21 23 25 27 Scaling Times 0 11 22 33 44 55 T-sign Text-PGen Text-Selected Figure 1: The translation performance of back- translation when scaling the generated spoken language data from 1) our PGEN which uses the large pretrained language model, and 2) selected in-domain data from large-scale spoken language corpus. The red dashed line denotes the baseline model without back-translation. Best viewed in color. them, sign language translation (SLT) aims to trans- form continuous sign language videos into natural spoken language texts (Bungeroth and Ney, 2004; Camgoz et al., 2018). SLT consists of two sub- tasks: (1) sign language understanding task that recognizes the continued videos to the sign glosses; and (2) spoken language gloss translation …

Content truncated to 500 words