SMILE Swiss German Sign Language Dataset

SMILE Swiss German Sign Language Dataset Sarah Ebling∗, Necati Cihan Camg¨oz†, Penny Boyes Braem∗, Katja Tissi∗, Sandra Sidler-Miserez∗, Stephanie Stoll†, Simon Hadfield†, Tobias Haug∗, Richard Bowden†, Sandrine Tornay‡, Marzieh Razavi‡, Mathew Magimai-Doss‡, ∗University of Applied Sciences of Special Needs Education (HfH), Schaffhauserstrasse 239, 8050 Zurich, Switzerland {sarah.ebling, katja.tissi, tobias.haug}@hfh.ch, boyesbraem@fzgresearch.org, sandysidler@gmail.com †CVSSP, University of Surrey, GU2 7XR, Guildford, UK {n.camgoz, s.m.stoll, s.hadfield, r.bowden}@surrey.ac.uk ‡Idiap Research Institute, 1920 Martigny, Switzerland {sandrine.tornay, marzieh.razavi, mathew}@idiap.ch Abstract Sign language recognition (SLR) involves identifying the form and meaning of isolated signs or sequences of signs. To our knowledge, the combination of SLR and sign language assessment is novel. The goal of an ongoing three-year project in Switzerland is to pioneer an assessment system for lexical signs of Swiss German Sign Language (Deutschschweizerische Geb¨ardensprache, DSGS) that relies on SLR. The assessment system aims to give adult L2 learners of DSGS feedback on the correctness of the manual parameters (handshape, hand position, location, and movement) of isolated signs they produce. In its initial version, the system will include automatic feedback for a subset of a DSGS vocabulary production test consisting of 100 lexical items. To provide the SLR component of the assessment system with sufficient training samples, a large-scale dataset containing videotaped repeated productions of the 100 items of the vocabulary test with associated transcriptions and annotations was created, consisting of data from 11 adult L1 signers and 19 adult L2 learners of DSGS. This paper introduces the dataset, which will be made available to the research community. Keywords: Swiss German Sign Language (DSGS), automatic sign language assessment, sign language testing, sign language recognition and analysis, Microsoft Kinect v2, GoPro, L2 acquisition 1 Introduction Swiss German Sign Language (Deutschschweizerische Geb¨ardensprache, DSGS) has approximately 5,500 Deaf1 L1 users. In addition, an estimated 13,000 hearing persons use DSGS. Among them are children of deaf adults (CO- DAs), sign language interpreters, teachers, social workers, and persons otherwise interested in the language (Boyes Braem, 2012). With the exception of CODAs, they are of- ten adult L2 learners of DSGS. DSGS is composed of five dialects that originated in former schools for the Deaf. The differences between the dialects are primarily lexical and pertain, e.g., to semantic fields such as food (distinct signs for regional food items, such as specific breads) and date specifications (distinct signs for weekdays and months) (Boyes Braem, 1983). The goal of the ongoing three-year SMILE (Scalable Mul- timodal Sign Language Technology for Sign Language Learning and Assessment) project in Switzerland is to pi- oneer an assessment system for lexical signs of DSGS that relies on sign language recognition (SLR) technology. SLR involves identifying the form and meaning of isolated signs or sequences of signs. While SLR has been applied to sign language learning (Spaai et al., 2005; Huenerfauth et al., 2017), to our knowledge, the combination of SLR and sign language assessment is novel. The assessment system that is being developed as part of the SMILE project aims to give adult L2 learners of DSGS 1It is …

Content truncated to 500 words