A Low-Cost Motion Capture Corpus in French Sign Language for Interpreting Iconicity and Spatial Referencing Mechanisms

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2488–2497 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2488 A Low-Cost Motion Capture Corpus in French Sign Language for Interpreting Iconicity and Spatial Referencing Mechanisms Cl´emence Mertz∗, Vincent Barreaud∗, Thibaut Le Naour†, Damien Lolive∗, Sylvie Gibet‡ ∗Universit´e Rennes1, IRISA, France clemence.mertz, vincent.barreaud, damien.lolive@irisa.fr †Motion-Up, France contact@motion-up.com ‡Universit´e Bretagne Sud, IRISA, France sylvie.gibet@irisa.fr Abstract The automatic translation of sign language videos into transcribed texts is rarely approached in its whole, as it implies to finely model the grammatical mechanisms that govern these languages. The presented work is a first step towards the interpretation of French sign language (LSF) by specifically targeting iconicity and spatial referencing. This paper describes the LSF-SHELVES corpus as well as the original technology that was designed and implemented to collect it. Our final goal is to use deep learning methods to circumvent the use of models in spatial referencing recognition. In order to obtain training material with sufficient variability, we designed a light-weight (and low-cost) capture protocol that enabled us to collect data from a large panel of LSF signers. This protocol involves the use of a portable device providing a 3D skeleton, and of a software developed specifically for this application to facilitate the post-processing of handshapes. The LSF-SHELVES includes simple and compound iconic and spatial dynamics, organized in 6 complexity levels, representing a total of 60 sequences signed by 15 LSF signers. Keywords: Corpus, Motion Capture, French Sign Language, Kinect Azure, MotionUp software 1. Introduction Assistance in learning French sign language (LSF) re- quires the creation of appropriate resources and the de- velopment of digital interfaces that facilitate access to information. However, main existing data are limited lexicons, a few narrations or dialogues in LSF, and very few are publicly available and labeled with transcribed texts. Furthermore, corpora dedicated to grammatical structures in LSF remain almost non-existent. Our goal is to design and develop digital and pedagog- ical tools that facilitate the learning (or reinforcement) of LSF, focusing on well identified grammatical mech- anisms. With the increase in the amount of data and the advent of new machine learning techniques, these tools should assist the automatized translation from videos in LSF into written transcriptions in French. Since it is not possible to tackle the complete machine translation pipeline, given the successive transforma- tions, from video to 3D motion, and from 3D motion to written LSF, we focus in this work on a partial mod- eling and interpretation system that makes possible the interpretation of signed sentences. This system is char- acterized by 1) a simplified methodology of the trans- lation process; 2) the selection of limited LSF gram- matical mechanisms; 3) the design of an original data acquisition technique that facilitates the capture, stor- age and post-processing of large volumes of data. The machine translation process can be described as a dual translation process, in which the signed and la- beled videos (the subtitles being usually …

Content truncated to 500 words