A Dataset for Linguistic Understanding, Visual Evaluation, and Recognition of Sign Languages: The K-RSL

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 631–640 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 631 K-RSL: a Corpus for Linguistic Understanding, Visual Evaluation, and Recognition of Sign Languages Alfarabi Imashev, Medet Mukushev, Vadim Kimmelman†, Anara Sandygulova Department of Robotics and Mechatronics, School of Engineering and Digital Sciences, Nazarbayev University, Kazakhstan †Department of Linguistic, Literary and Aesthetic Studies, University of Bergen, Norway alfarabi.imashev@nu.edu.kz, mmukushev@nu.edu.kz, vadim.kimmelman@uib.no, anara.sandygulova@nu.edu.kz Abstract The paper presents the first dataset that aims to serve interdisciplinary purposes for the utility of computer vision community and sign lan- guage linguistics. To date, a majority of Sign Language Recognition (SLR) approaches fo- cus on recognising sign language as a manual gesture recognition problem. However, sign- ers use other articulators: facial expressions, head and body position and movement to con- vey linguistic information. Given the impor- tant role of non-manual markers, this paper proposes a dataset and presents a use case to stress the importance of including non-manual features to improve the recognition accuracy of signs. To the best of our knowledge no prior publicly available dataset exists that ex- plicitly focuses on non-manual components re- sponsible for the grammar of sign languages. To this end, the proposed dataset contains 28250 videos of signs of high resolution and quality, with annotation of manual and non- manual components. We conducted a series of evaluations in order to investigate whether non-manual components would improve signs’ recognition accuracy. We release the dataset to encourage SLR researchers and help ad- vance current progress in this area toward real- time sign language interpretation. Our dataset will be made publicly available at https:// krslproject.github.io/krsl-corpus 1 Introduction There exist over 300 sign languages around the world that are native to 70 million deaf people (Bragg et al., 2019). Sign languages are com- prised of hand gestures, arms and body movements, head position, facial expressions, and lip patterns (Sandler and Lillo-Martin, 2006). While auto- matic speech recognition has progressed to being commercially available, automatic Sign Language Recognition (SLR) is still in its infancy (Cooper et al., 2011). To date, more than half of published vision- based research utilizes isolated sign language data with a vocabulary size of less than 50 signs (Koller, 2020). But the real-world utility of SLR solutions requires continuous recognition, which is signifi- cantly more challenging than recognising individ- ual signs due to co-articulation (the ending of one sign affecting the start of the next), depiction (visu- ally representing or enacting content), epenthesis effects (insertion of extra features into signs), gen- eralization, and so on (Bragg et al., 2019). As a result, realistic, generalisable, and large datasets are necessary to advance SLR. Current efforts in SLR do not address the com- plexities of sign language linguistics, and thus have a limited real-world value (Bragg et al., 2019). Chatzis et al. (2020) highlight the importance of non-manual components of sign languages. For example, they can change meaning of a verb, or differentiate between objects and people. Accord- ing to Koller …

Content truncated to 500 words