A Salience-Based Approach to Gesture-Speech Alignment

A Salience-Based Approach to Gesture-Speech Alignment Jacob Eisenstein and C. Mario Christoudias MIT Computer Science and Artificial Intelligence Laboratory 32 Vassar Street Cambridge, MA 02139 {jacobe+cmch}@csail.mit.edu Abstract One of the first steps towards understanding natural multimodal language is aligning ges- ture and speech, so that the appropriate ges- tures ground referential pronouns in the speech. This paper presents a novel technique for gesture-speech alignment, inspired by salience- based approaches to anaphoric pronoun reso- lution. We use a hybrid between data-driven and knowledge-based mtehods: the basic struc- ture is derived from a set of rules about gesture salience, but the salience weights themselves are learned from a corpus. Our system achieves 95% recall and precision on a corpus of tran- scriptions of unconstrained multimodal mono- logues, significantly outperforming a competi- tive baseline. 1 Introduction In face to face communication, speakers frequently use gesture to supplement speech (Chovil, 1992), using the additional modality to provide unique, non-redundant in- formation (McNeill, 1992). In the context of pen/speech user interfaces, Oviatt finds that “multimodal ... language is briefer, syntactically simpler, and less disfluent than users’ unimodal speech.” (Oviatt, 1999) One of the simplest and most direct ways in which ges- ture can supplement verbal communication is by ground- ing references, usually through deixis. For example, it is impossible to extract the semantic content of the verbal utterance “I’ll take this one” without an accompanying pointing gesture indicating the thing that is desired. The problem of gesture-speech alignment involves choosing the appropriate gesture to ground each verbal utterance. This paper describes a novel technique for this problem. We evaluate our system on a corpus of multimodal mono- logues with no fixed grammar or vocabulary. 1.1 Example [This]_1 thing goes over [here]_2 so that it goes back ... -------------------- 1. Deictic: Hand rests on latch mechanism 2. Iconic: Hand draws trajectory from right to left In this example, there are three verbal references. The word “this” refers to the latch mechanism, which is indi- cated by the rest position of the hand. “Here” refers to the endpoint of the trajectory indicated by the iconic gesture. “It” is an anaphoric reference to a noun phrase defined earlier in the sentence; there is no accompanying gesture. The word “that” does not act as a reference, although it could in other cases. Not every pronoun keyword (e.g., this, here, it, that, etc.) will act as a reference in all cases. In addition, there will be many gestures that do not re- solve any keyword. 2 Related Work This research draws mainly from two streams of re- lated work. Researchers in human-computer interaction have worked towards developing multimodal user inter- faces, which allow spoken and gestural input. These sys- tems often feature powerful algorithms for fusing modal- ities; however, they also restrict communication to short grammatically-constrained commands over a very lim- ited vocabulary. Since our goal is to handle more com- plex linguistic phenomena, these systems were of little help in the design of our algorithm. Conversely, we found that …

Content truncated to 500 words