Automatic Gloss-level Data Augmentation for Sign Language Translation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6808–6813 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6808 Automatic Gloss-level Data Augmentation for Sign Language Translation Jin Yea Jang⋆,§,Hanmu Park§,Saim Shin§,Suna Shin†,Byungcheon Yoon‡,Gahgene Gweon⋆ ⋆Department of Intelligence and Information, Seoul National University §AIRC, Korea Electronics Technology Institute †Sign Language Linguistics Laboratory, Korea Nazarene University ‡Department of Sign Language Interpreting, Korea Nazarene University ⋆1 Gwanak-ro, Gwanak-gu, Seoul , Republic of Korea §22 Daewangpangyo-ro 712beon-gil, Bundang-gu, Seongnam-si, Gyeonggi-do, Republic of Korea †,‡48 Wolbong-ro, Seobuk-gu, Cheonan-si, Chungcheongnam-do, Republic of Korea {jinyea.jang, ggweon}@snu.ac.kr,{hanmu,sishin}@keti.re.kr,ksl power@daum.net,ybch5778@kornu.ac.kr Abstract Securing sufficient data to enable automatic sign language translation modeling is challenging. The data insufficiency issue exists in both video and text modalities; however, fewer studies have been performed on text data augmentation compared to video data. In this study, we present three methods of augmenting sign language text modality data, comprising 3,052 Gloss-level Korean Sign Language (GKSL) and Word-level Korean Language (WKL) sentence pairs. Using each of the three methods, the following number of sentence pairs were created: blank replacement 10,654, sentence paraphrasing 1,494, and synonym replacement 899. Translation experiment results using the augmented data showed that when translating from GKSL to WKL and from WKL to GKSL, Bi-Lingual Evaluation Understudy (BLEU) scores improved by 0.204 and 0.170 respectively, compared to when only the original data was used. The three contributions of this study are as follows. First, we demonstrated that three different augmentation techniques used in existing Natural Language Processing (NLP) can be applied to sign language. Second, we propose an automatic data augmentation method which generates quality data by utilizing the Korean sign language gloss dictionary. Lastly, we publish the Gloss-level Korean Sign Language 13k dataset (GKSL13k), which has verified data quality through expert reviews. Keywords: sign language translation, sign language gloss, data augmentation 1. Introduction Research on automatic sign language translation tech- nologies to improve accessibility for the hearing im- paired is being actively conducted (Camgoz et al., 2018; Ko et al., 2019; Miyazaki et al., 2020). A two- stage model, which is shown in Figure 1, is a popular translation model. The first stage recognizes sign lan- guage gloss (Ormel et al., 2010) units (Sincan and Ke- les, 2020; Imashev et al., 2020), and the second stage translates gloss sequences into target-language sen- tences. The two-stage model has two main advantages. First, such a model can alleviate the out-of-vocabulary problem when using a separate sign language gloss dic- tionary. Second, a two-stage model is flexible, in that various NLP techniques can be applied independent of the first stage. To ensure adequate performance of the two-stage model, sufficient data, especially gloss-level text data is required. However, accumulating sufficient sign lan- guage data is challenging due to the lack of expert- annotated datasets. Herein, we present new methods for automatically augmenting translation data on Gloss- level Korean Sign Language (GKSL) and Word-level Korean Language (WKL) sentence pairs. Gloss-level data augmentation methods are rare. A re- lated …

Content truncated to 500 words