Evaluation of Manual and Non-manual Components for Sign Language Recognition

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6073–6078 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6073 Evaluation of Manual and Non-manual Components for Sign Language Recognition Medet Mukushev∗, Arman Sabyrov∗, Alfarabi Imashev∗, Kenessary Koishybay∗ Vadim Kimmelman†, Anara Sandygulova∗‡ ∗Department of Robotics and Mechatronics, School of Engineering and Digital Sciences, Nazarbayev University Kabanbay Batyr Avenue, 53, Nur-Sultan, Kazakhstan †Department of Linguistic, Literary and Aesthetic Studies, University of Bergen Postboks 7805, 5020, Bergen, Norway mmukushev@nu.edu.kz, arman.sabyrov@nu.edu.kz, alfarabi.imashev@nu.edu.kz, kenessary.koishybay@nu.edu.kz vadim.kimmelman@uib.no, anara.sandygulova@nu.edu.kz Abstract The motivation behind this work lies in the need to differentiate between similar signs that differ in non-manual components present in any sign. To this end, we recorded full sentences signed by five native signers and extracted 5200 isolated sign samples of twenty frequently used signs in Kazakh-Russian Sign Language (K-RSL), which have similar manual components but differ in non-manual components (i.e. facial expressions, eyebrow height, mouth, and head orientation). We conducted a series of evaluations in order to investigate whether non-manual components would improve sign’s recognition accuracy. Among standard machine learning approaches, Logistic Regression produced the best results, 78.2% of accuracy for dataset with 20 signs and 77.9% of accuracy for dataset with 2 classes (statement vs question). Dataset can be downloaded from the following website: https://krslproject.github.io/krsl20/ Keywords: Sign language Recognition, Information extraction, Statistical and machine learning methods 1. Introduction Deaf communities around the world communicate via sign languages to express meaning and intent (Sandler and Lillo- Martin, 2006). Similar to spoken languages, each country or region has its own sign language of varying grammar and rules, leading to a few hundreds of sign languages that exist today. While automatic speech recognition has pro- gressed to being commercially available, automatic Sign Language Recognition (SLR) is still in its infancy (Cooper et al., 2011). Most works on sign language recognition consider man- ual and non-manual components separately. Manual fea- tures are features related to hands (e.g. hand configura- tion and motion trajectory of hands), while non-manual fea- tures include facial expressions, gaze direction, lip patterns, head and body posture. For example, signers use articula- tors such as facial expressions, head and body position and movement to convey linguistic information (Pfau and Quer, 2010). It has been shown that non-manual markers function at different levels in sign languages. On the lexical level, signs which are manually identical can be distinguished by facial expression or specifically by mouthing (silent articu- lation of a word from a spoken language) (Crasborn et al., 2008). On the morphological level, facial expressions and mouth patterns are used to convey adjectival and adverbial information (e.g. indicate the size of objects or aspectual properties of events) (Crasborn et al., 2008). Non-manual markers are especially important on the sentence level and beyond. Almost universally, the negation in many sign lan- guages is expressed by head movements (Zeshan, 2004a), while questions are distinguished from statements by eye- brow and head position (Zeshan, 2004b). Given the important role of non-manual …

Content truncated to 500 words