Detection of Major ASL Sign Types in Continuous Signing For ASL Recognition

Detection of Major ASL Sign Types in Continuous Signing for ASL Recognition Polina Yanovich, Carol Neidle, Dimitris Metaxas Rutgers, The State University of New Jersey, Computer Science Boston University, Linguistics Program 110 Frelinghuysen Rd., Piscataway NJ, 08854 621 Commonwealth Ave., Boston, MA 02215 yanovich@cs.rutgers.edu, carol@bu.edu, dnm@cs.rutgers.edu Abstract In American Sign Language (ASL) as well as other signed languages, different classes of signs (e.g., lexical signs, fingerspelled signs, and classifier constructions) have different internal structural properties. Continuous sign recognition accuracy can be improved through use of distinct recognition strategies, as well as different training datasets, for each class of signs. For these strategies to be applied, continuous signing video needs to be segmented into parts corresponding to particular classes of signs. In this paper we present a multiple instance learning-based segmentation system that accurately labels 91.27% of the video frames of 500 continuous utterances (including 7 different subjects) from the publicly accessible NCSLGR corpus <http://secrets.rutgers.edu/dai/queryPages/> (Neidle and Vogler, 2012). The system uses novel feature descriptors derived from both motion and shape statistics of the regions of high local motion. The system does not require a hand tracker. Keywords: continuous signing, ASL recognition, sign classification 1. Introduction Computer-based ASL recognition often focuses on signs that have been pre-extracted from video (with known start and end frames) (Elons et al., 2013a; Mohandes et al., 2012, e.g.). Learning methods are typically employed for sign recognition; these are frequently based on Hidden Markov Models (HMMs) (Vogler and Metaxas, 2004) or Conditional Random Fields (CRFs) (Wang et al., 2006), trained for each sign in the vocabulary. However, continuous signing presents a significantly greater challenge, not only because of co-articulation ef- fects, but also because of the existence of multiple classes of signs with fundamentally different internal composition. Unlike spoken languages, in which phonemes are con- catenated, in signed languages the components of words combine both sequentially and non-sequentially (simulta- neously), subject to different linguistic constraints depend- ing on the morphological class to which the signs belong, and these linguistic constraints can be leveraged to im- prove computer-based recognition (Athitsos et al., 2010; Thangali et al., 2011). The three most prevalent classes in signed languages are lexical signs, fingerspelled signs, and classifier constructions. Lexical sign production involves combinations of spe- cific hand configurations, orientations, locations in sign- ing space, and movement trajectories. Strict linguistic constraints govern the relationships between start and end handshapes of a given sign, and between the two hands (in 2-handed signs) with respect to hand configuration and movement trajectory (Battison, 1978; Brentari, 1998; Thangali et al., 2011). Fingerspelled signs consist of sequences of letter hand- shapes from the manual alphabet, produced with rapid fin- ger movements at a fairly constant global hand location (Athitsos et al., 2010), potentially with relatively small left- to-right movement. Although some fingerspelled signs are in frequent usage, fingerspelling is often used for proper names and spoken language borrowings. Thus there is no fixed fingerspelled vocabulary: many fingerspelled produc- tions would not be included in any ASL dictionary. Classifier …

Content truncated to 500 words