Dicta-Sign-LSF-v2: Remake of a Continuous French Sign Language Dialogue Corpus and a First Baseline for Automatic Sign Language Processing

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6040–6048 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6040 Dicta-Sign–LSF–v2: Remake of a Continuous French Sign Language Dialogue Corpus and a First Baseline for Automatic Sign Language Processing Valentin Belissen1, Annelies Braffort2, Michèle Gouiffès3 1, 2, 3LIMSI–CNRS, 1, 3Université Paris Sud, 1, 3Université Paris Saclay 1, 2, 3LIMSI, Campus universitaire 507, Rue du Belvedère, 91405 Orsay - France {valentin.belissen, annelies.braffort, michele.gouiffes}@limsi.fr Abstract While the research in automatic Sign Language Processing (SLP) is growing, it has been almost exclusively focused on recognizing lexical signs, whether isolated or within continuous SL production. However, Sign Languages include many other gestural units like iconic structures, which need to be recognized in order to go towards a true SL understanding. In this paper, we propose a newer version of the publicly available SL corpus Dicta-Sign, limited to its French Sign Language part. Involving 16 different signers, this dialogue corpus was produced with very few constraints on the style and content. It includes lexical and non-lexical annotations over 11 hours of video recording, with 35000 manual units. With the aim of stimulating research in SL understanding, we also provide a baseline for the recognition of lexical signs and non-lexical structures on this corpus. A very compact modeling of a signer is built and a Convolutional-Recurrent Neural Network is trained and tested on Dicta-Sign–LSF–v2, with state-of-the-art results, including the ability to detect iconicity in SL production. Keywords: Sign Language Recognition, Sign Language Processing, Iconicity, Depicting Signs, French Sign Language 1. Introduction A great number of sign languages have been naturally developed within Deaf communities, and even the most widely-spoken are still to be thoroughly described in terms of linguistics (Braffort, 2016). They all have their own number of very conventionalized signs as well as highly iconic structures, such as classifiers constructions (Liddell, 1977) used to depict common entities (see Fig. 1). While the research in automatic Sign Language Processing (SLP) has focused on recognizing lexical signs which are by def- inition very conventionalized, iconicity is usually ignored even though it is crucial for SL understanding. In this paper, three major contributions to the field of con- tinuous SLP are presented: 1. A public remake of the French Sign Language (LSF) part of the Dicta-Sign Corpus (Matthes et al., 2012), with cleaned and reliable annotations. These annota- tions include lexical data and more refined linguistic categories. As the corpus is based on dialogue with very loose elicitation guidelines, it is highly represen- tative of natural SL. 2. The manufacturing of a relevant modeling of a signer as a proper input to a convolutional-recurrent neural network (CRNN) for automatic SLP. This modeling is made compact, and easily generalizable to any SL recording. Preprocessed data is also made public; 3. We prove the relevance of our modeling and network on Dicta-Sign–LSF–v2. State-of-the-art accuracy for the classification of video frames into a number of manual unit types is also attained on NCSLGR, …

Content truncated to 500 words