Sign Language Recognition with Transformer Networks

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6018–6024 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6018 Sign Language Recognition with Transformer Networks Mathieu De Coster1, Mieke Van Herreweghe2, Joni Dambre1 1IDLab-AIRO – Ghent University - imec, 2Ghent University 1Technologiepark-Zwijnaarde 126, Ghent, Belgium, 2Blandijnberg 2, Ghent, Belgium {mathieu.decoster, mieke.vanherreweghe, joni.dambre}@ugent.be Abstract Sign languages are complex languages. Research into them is ongoing, supported by large video corpora of which only small parts are annotated. Sign language recognition can be used to speed up the annotation process of these corpora, in order to aid research into sign languages and sign language recognition. Previous research has approached sign language recognition in various ways, using feature extraction techniques or end-to-end deep learning. In this work, we apply a combination of feature extraction using OpenPose for human keypoint estimation and end-to-end feature learning with Convolutional Neural Networks. The proven multi-head attention mechanism used in transformers is applied to recognize isolated signs in the Flemish Sign Language corpus. Our proposed method significantly outperforms the previous state of the art of sign language recognition on the Flemish Sign Language corpus: we obtain an accuracy of 74.7% on a vocabulary of 100 classes. Our results will be implemented as a suggestion system for sign language corpus annotation. Keywords: sign language recognition, deep learning, corpus annotation 1. Introduction Sign language recognition (SLR) is a complex problem. Sign languages are, after all, complex visual languages. Generally, one can say that sign languages have five parameters. A sign is distinguished by hand shape, hand orientation, movement, location, and non-manual components such as mouth shape and eyebrow shape. However, these parameters do not necessarily fully identify signs: two signs can have the same execution but different meaning. Furthermore, identical signs are often executed differently based on several factors, such as age, gender, the dominant hand, and dialects. Additionally, there is a high degree of co-articulation in sign languages: both hands can produce different signs at the same time. SLR is typically tackled using machine learning approaches. Deep learning in particular has proven to be very powerful for tasks such as image classification (Krizhevsky et al., 2012) and neural translation (Vaswani et al., 2017). Deep learning algorithms require large datasets in order to learn meaningful representations that generalize well to unseen data, especially for complex problems such as SLR. While large video corpora are available for several sign languages, they consist mostly of unlabeled data. Labeling the sign language corpora is a time-consuming process that requires the annotator to know sign language and its specific phonetic and phonological properties. As a consequence, the portion of a video corpus that is labeled grows only slowly. Larger datasets exist (Chai et al., 2014; Huang et al., 2018; Vaezi Joze and Koller, 2019), but several consist of recordings of persons performing signs in repetition (Ronchetti et al., 2016; Ko et al., 2018). These datasets are often not representative of real world sign language, as they …

Content truncated to 500 words