An HMM Approach with Inherent Model Selection for Sign Language and Gesture Recognition

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6049–6056 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6049 An HMM Approach with Inherent Model Selection for Sign Language and Gesture Recognition Sandrine Tornay, Oya Aran, Mathew Magimai.-Doss Idiap Research Institute, Ecole Polytechnique F´ed´erale de Lausanne, De La Salle Univerity Martigny CH-1920 Switzerland, Lausanne CH-1015 Switzerland, Malate Manila Philippines {sandrine.tornay, mathew}@idiap.ch Abstract HMMs have been the one of the first models to be applied for sign recognition and have become the baseline models due to their success in modeling sequential and multivariate data. Despite the extensive use of HMMs for sign recognition, determining the HMM structure has still remained as a challenge, especially when the number of signs to be modeled is high. In this work, we present a continuous HMM framework for modeling and recognizing isolated signs, which inherently performs model selection to optimize the number of states for each sign separately during recognition. Our experiments on three different datasets, namely, German sign language DGS dataset, Turkish sign language HospiSign dataset and Chalearn14 dataset show that the proposed approach achieves better sign language or gesture recognition systems in comparison to the approach of selecting or presetting the number of HMM states based on k-means, and yields systems that perform competitive to the case where the number of states are determined based on the test set performance. Keywords: Sign Language Recognition, Hidden Markov Models, Model Selection 1. Introduction Following the recent developments and advancements on automatic speech recognition, commercial systems are be- coming available in our daily life, where users can use their everyday communication means, i.e. speech, to interact with machines. However these systems rarely address the deaf members of our society, who predominantly use sign languages as their primary means of communication. For more accessible technology, research on automatic Sign Language Recognition (SLR) should also advance, which is currently in its infancy. Sign languages are the natural communication media of deaf people. In spoken languages, the words are pro- duced through the vocal tract and are perceived as sounds, whereas in sign languages, the signs are produced alone or simultaneously, by use of hand shape, hand motion, hand location, as well as facial expression, head motion, and body posture, and they are perceived visually. The pro- duction of signs follow both sequential and parallel nature: signs come one after the other showing a sequential behav- ior; at the same time each sign may contain parallel actions of hands, face, head or body (Stokoe, 2005). Due to its vi- sual nature, the multimodal properties, and parallel use of different channels, sign languages present other challenges for automatic recognition, from feature extraction to mod- eling, in comparison to spoken languages. HMMs offer a natural solution for SLR with their power in handling sequential and multimodal data. They are exten- sively used and have proven successful in the sign language recognition domain (Ong and Ranganath, 2005; Cooper et al., 2011). One of …

Content truncated to 500 words