Continuous Profile Models in ASL Syntactic Facial Expression Synthesis

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2084–2093, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Continuous Profile Models in ASL Syntactic Facial Expression Synthesis Hernisa Kacorri Carnegie Mellon University Human-Computer Interaction Institute 5000 Forbes Avenue Pittsburgh, PA 15213, USA hkacorri@andrew.cmu.edu Matt Huenerfauth Rochester Institute of Technology B. Thomas Golisano College of Computing and Information Sciences 152 Lomb Memorial Drive Rochester, NY 14623, USA matt.huenerfauth@rit.edu Abstract To create accessible content for deaf users, we investigate automatically synthesizing animations of American Sign Language (ASL), including grammatically important facial expressions and head movements. Based on recordings of humans perform- ing various types of syntactic face and head movements (which include idiosyn- cratic variation), we evaluate the efficacy of Continuous Profile Models (CPMs) at identifying an essential “latent trace” of the performance, for use in producing ASL animations. A metric-based evalua- tion and a study with deaf users indicated that this approach was more effective than a prior method for producing animations. 1 Introduction and Motivation While there is much written content online, many people who are deaf have difficulty reading text or may prefer sign language. For example, in the U.S., standardized testing indicates that a major- ity of deaf high school graduates (age 18+) have a fourth-grade reading level or below (Traxler, 2000) (U.S. fourth-grade students are typically age 9). While it is possible to create video-recordings of a human performing American Sign Language (ASL) for use on websites, updating such material is expensive (i.e., re-recording). Thus, researchers investigate technology to automate the synthesis of animations of a signing virtual human, to make it more cost-effective for organizations to provide sign language content online that is easily updated and maintained. Animations can be automatically synthesized from a symbolic specification of the message authored by a human or perhaps by ma- chine translation, e.g. (Ebling and Glauert, 2013; Filhol et al., 2013; Stein et al., 2012). 1.1 ASL Syntactic Facial Expressions Facial expressions are essential in ASL, conveying emotion, semantic variations, and syntactic struc- ture. Prior research has verified that ASL ani- mations with missing or poor facial expressions are significantly less understandable for deaf users (Kacorri et al., 2014; Kacorri et al., 2013b; Ka- corri et al., 2013a). While artists can produce indi- vidual animations with beautiful expressions, such work is time-consuming. For efficiently maintain- able online content, we need automatic synthesis of ASL from a sparse script representing the lexi- cal items and basic elements of the sentence. Specifically, we are studying how to model and generate ASL animations that include syntactic facial expressions, conveying grammatical infor- mation during entire phrases and therefore con- strained by the timing of the manual signs in a phrase (Baker-Shenk, 1983). Generally speaking, in ASL, upper face movements (examined in this paper) convey syntactic information across entire phrases, with the mouth movements conveying lexical or adverbial information. The meaning of a sequence of signs performed with the hands depends on the co-occuring fa- cial expression. (While …

Content truncated to 500 words