MEDIAPI-SKEL - A 2D-Skeleton Video Database of French Sign Language With Aligned French Subtitles

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6063–6068 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6063 MEDIAPI-SKEL - A 2D-Skeleton Video Database of French Sign Language With Aligned French Subtitles Hannah Bull1, Annelies Braffort2, Mich`ele Gouiff`es3 1,2,3LIMSI-CNRS, 1,3University of Paris-Saclay 1,2,3LIMSI, Campus universitaire 507, Rue du Belved`ere, 91405 Orsay - France {hannah.bull, annelies.braffort, michele.gouiffes}@limsi.fr Abstract This paper presents MEDIAPI-SKEL, a 2D-skeleton database of French Sign Language videos aligned with French subtitles. The corpus contains 27 hours of video of body, face and hand keypoints, aligned to subtitles with a vocabulary size of 17k tokens. In contrast to existing sign language corpora such as videos produced under laboratory conditions or translations of TV programs into sign language, this database is constructed using original sign language content largely produced by deaf journalists at the media company M´edia-Pi. Moreover, the videos are accurately synchronized with French subtitles. We propose three challenges appropriate for this corpus that are related to processing units of signs in context: automatic alignment of text and video, semantic segmentation of sign language, and production of video-text embeddings for cross-modal retrieval. These challenges deviate from the classic task of identifying a limited number of lexical signs in a video stream. Keywords: Sign Language Processing, Computer Vision, Alignment, Retrieval, Video Embeddings 1. Introduction There is a relative lack of sign language corpora in com- parison to other areas of natural language processing, par- ticularly of large, diverse sign language corpora with high quality native speakers in natural settings. Moreover, much attention in the computer vision literature has been given to automatic detection of a limited number of signs, in com- parison to other sign language processing tasks (Bragg et al., 2019). In order to combat these two shortcomings, we propose a new dataset for new challenges. We provide a new corpus available for public research called MEDIAPI-SKEL1 (Bull, Hannah and Braffort, An- nelies, 2019). Our dataset consists of 368 videos total- ing 27 hours of French Sign Language (LSF) with French subtitles, generated from the content of the bilingual LSF- French media company M´edia-Pi2. The videos are pro- vided in the form of 2D-skeletons with 135 face, hand and body keypoints, but the original videos can be accessed through a subscription with M´edia-Pi. The subtitles pro- vide an accurate alignment between short segments of text and short segments of sign language video. A frame of this data is shown in Figure 1. This new corpus allows for challenges for sign language processing at a ‘sentence’ or ‘phrase’ level, rather than at the ‘word’ or ‘sign’ level. We propose three such machine learning challenges for MEDIAPI-SKEL. The structure of the article is as follows. Firstly we in- troduce the unique features of sign languages and the re- sulting limitations of the lexical approach to sign language processing. Secondly, we discuss differences between MEDIAPI-SKEL and other existing sign language corpora. Thirdly, we justify our particular focus on 2D-skeleton data. Fourthly, we …

Content truncated to 500 words