Pipeline Signed Japanese Translation Focusing on a Post-positional Particle Complement and Conjugation in a Low-resource Setting

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2021–2032 August 1–6, 2021. ©2021 Association for Computational Linguistics 2021 Pipeline Signed Japanese Translation Focusing on a Post-positional Particle Complement and Conjugation in a Low-resource Setting Ken Yano and Akira Utsumi The University of Electro-Communications 1-5-1 Chofugaoka, Chofu, Tokyo, JAPAN yanoken@uec.ac.jp, utsumi@uec.ac.jp Abstract Because sign language is a visual language, the translation of it into spoken language is typically performed through an intermediate representation called gloss notation. In sign language, function words, such as particles and determiners, are not explicitly expressed, and there is little or no concept of morpho- logical inflection in sign language. Therefore, gloss notation does not include such linguis- tic constructs. Because of these factors, we argue that sign language translation is effec- tively processed by taking advantage of the similarities and differences between sign lan- guage and its spoken counterpart. We thus propose a pipeline translation method that clearly focuses on the difference between spo- ken Japanese and signed Japanese written in gloss notation. Specifically, our method first uses statistical machine translation (SMT) to map glosses to corresponding spoken language words. We then use three transformer-based seq2seq models trained using a large out-of- domain monolingual Japanese corpus to com- plement postpositional particles and estimate conjugations for the verbs, adjectives, and aux- iliary verbs in the first translation. We ap- ply the seq2seq models in sequence until the translation converges. Our experimental re- sults show that the proposed method performs robustly on the low-resource corpus and is +4.4/+4.9 points above the SMT baseline for BLEU-3/4. 1 Introduction It is essential to build a social infrastructure for hearing-impaired and hearing communities to share sufficient information so that they can quickly obtain information necessary for daily life and disasters and lead a safe and secure life. Sign language used in deaf communities has differ- ent vocabulary and grammar from spoken lan- guage. There are two variations of sign language in Japan (Chonan, 2001): (1) Japanese Sign Lan- guage (JSL) and (2) Manually Coded Japanese (MCJ). JSL is often used by early signers, and syntax, such as word order and language struc- ture, is different from spoken Japanese. By con- trast, the syntax of MCJ is similar to that of spo- ken Japanese in terms of word order. It is used by late signers or acquired hearing-impaired peo- ple. However, the two variations are said to be used interchangeably and there is no clear bound- ary between them. In this work, we consider an intermediate between JSL and MCJ, and denote it Signed Japanese (SJ) in the following discussion. Translation from sign language to spoken lan- guage is typically performed in two steps. First, consecutive signs are recognized from a video sig- nal and transformed into an intermediate represen- tation called a gloss, then the gloss is translated into a sentence in spoken language. Current state- of-the-art sign language recognition and transla- tion methods (Camg¨oz et al. 2020; Yin and Read 2020) require a large amount of data and …

Content truncated to 500 words