Linguistically-driven Framework for Computationally Efficient and Scalable Sign Recognition

Linguistically-driven Framework for Computationally Efficient and Scalable Sign Recognition Dimitris Metaxas*, Mark Dilsizian*, Carol Neidle** *Rutgers University, **Boston University *Rutgers University, CBIM, Department of Computer Science, 617 Bowser Road, Piscataway, NJ 08854 **Boston University Linguistics, 621 Commonwealth Ave., Boston, MA 02215 dnm@cs.rutgers.edu, mdil@cs.rutgers.edu, carol@bu.edu Abstract We introduce a new general framework for sign recognition from monocular video using limited quantities of annotated data. The novelty of the hybrid framework we describe here is that we exploit state-of-the art learning methods while also incorporating features based on what we know about the linguistic composition of lexical signs. In particular, we analyze hand shape, orientation, location, and motion trajectories, and then use CRFs to combine this linguistically significant in- formation for purposes of sign recognition. Our robust modeling and recognition of these sub-components of sign produc- tion allow an efficient parameterization of the sign recognition problem as compared with purely data-driven methods. This parameterization enables a scalable and extendable time-series learning approach that advances the state of the art in sign recognition, as shown by the results reported here for recognition of isolated, citation-form, lexical signs from Ameri- can Sign Language (ASL). Keywords: Sign Recognition, Model-based Machine Learning, Computer Vision, American Sign Language (ASL). 1. Introduction Automatic sign recognition is a difficult problem given the complexity of the linguistic structures in sign languag- es and the challenges in modeling 3D configurations and movements from 2D video. To address this problem, we use certain known linguistic properties of the language to structure the problem, and to inform, enhance, and correct visual recognition tasks. By combining these components into a unified optimization framework, we recognize isolated, citation-form lexical signs from American Sign Language (ASL) in a fully scalable manner. Whereas prior vision-based approaches to sign recogni- tion by computer had focused on detection of linguistical- ly important components, such as handshape and motion trajectory, neural networks have recently been applied to the overall problem of end-to-end sign recognition with- out attending to linguistic structure. The framework des- cribed here (1) exploits recent discriminative neural net- based learning approaches, coupled with generative model-based methods, to improve the detection and anal- ysis of the linguistically relevant components and fea- tures; (2) integrates knowledge of linguistic structures and dependencies to derive additional parameters; and (3) uses CRF learning methods to integrate these features for sign recognition. This enhances visual recognition capabilities for the critical sign components and offers a unified framework for sign recognition. This approach is successful working with limited quantities of annotated data and is scalable. 2. Previous Work Previous computer vision research on sign recognition has generally focused on aspects of sign production known to be linguistically important, including analysis of hand- shapes, upper body pose, and movement trajectories. Prior work on hand pose recognition in general includes Heap and Hogg (1996), Athitsos and Sclaroff (2001, 2003) and Tompson et al. (2014). Lu et al. (2003), Vogler and Metaxas (2004), and Isaacs and Foo (2004) focus specifically on the recognition of handshapes in sign …

Content truncated to 500 words