Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting

Findings of the Association for Computational Linguistics: EACL 2023, pages 1706–1724 May 2-6, 2023 ©2023 Association for Computational Linguistics Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting Zifan Jiang University of Zurich jiang@cl.uzh.ch Amit Moryossef Bar-Ilan University University of Zurich amitmoryossef@gmail.com Mathias Müller University of Zurich mmueller@cl.uzh.ch Sarah Ebling University of Zurich ebling@cl.uzh.ch Abstract This paper presents work on novel machine translation (MT) systems between spoken and signed languages, where signed languages are represented in SignWriting, a sign language writing system. Our work1 seeks to address the lack of out-of-the-box support for signed languages in current MT systems and is based on the SignBank dataset, which contains pairs of spoken language text and SignWriting con- tent. We introduce novel methods to parse, factorize, decode, and evaluate SignWriting, leveraging ideas from neural factored MT. In a bilingual setup—translating from American Sign Language to (American) English—our method achieves over 30 BLEU, while in two multilingual setups— translating in both direc- tions between spoken languages and signed languages—we achieve over 20 BLEU. We find that common MT techniques used to im- prove spoken language translation similarly af- fect the performance of sign language transla- tion. These findings validate our use of an in- termediate text representation for signed lan- guages to include them in NLP research. 1 Introduction Most current machine translation (MT) systems only support spoken language input and output (text or speech), which excludes around 200 dif- ferent signed languages used by up to 70 million deaf people2 worldwide from modern language technology. Since signed languages are also natu- ral languages, Yin et al. (2021) calls for including sign language processing (SLP) in natural language processing (NLP) research. From a technical point of view, SLP brings novel challenges to NLP due to the visual-gestural modal- ity of sign language and special linguistic features 1Code and documentation available at https://github. com/J22Melody/signwriting-translation 2According to the World Federation of the Deaf: https: //wfdeaf.org/our-work/ 15/06/2022, 14:49 Sign Translate https://sign.mt 1/1  Sign Translate Verse 41. He gave her his hand and helped her up. Then he called in the widows and all the believers, and he presented her to them alive. 137 / 500   Text      󲇈󵨑󸗨 󱦡 󰀡 󿌁 󲇡 󸗨 󲇸󲇐󶿈 󰁈 󳈠 󳈘 󺒲 󺒡 󲇡 󸗢 󲇸󲇐󶿈 󿌁    UNITED STATES UNITED KINGDOM FRANCE DETECT LANGUAGE ENGLISH FRENCH SPANISH Figure 1: Demo application based on our models, trans- lating from spoken languages to signed languages rep- resented in SignWriting, then to human poses. (e.g., the use of space, simultaneity, referencing), which requires both computer vision (CV) and NLP technologies. Crucially, the lack of a standardized or widely used written form for signed languages has hindered their inclusion in NLP research. However, sign language writing systems do exist and are sporadically used (e.g., SignWriting (Sut- ton, 1990) and HamNoSys (Prillwitz and Zienert, 1990)). Therefore, we adopt the proposal of Yin et al. (2021) to formulate the sign language trans- lation (SLT) …

Content truncated to 500 words