Gestural Cohesion for Topic Segmentation

Proceedings of ACL-08: HLT, pages 852–860, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Gestural Cohesion for Topic Segmentation Jacob Eisenstein, Regina Barzilay and Randall Davis Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology 77 Massachusetts Ave., Cambridge MA 02139 {jacobe, regina, davis}@csail.mit.edu Abstract This paper explores the relationship between discourse segmentation and coverbal gesture. Introducing the idea of gestural cohesion, we show that coherent topic segments are char- acterized by homogeneous gestural forms and that changes in the distribution of gestural features predict segment boundaries. Gestu- ral features are extracted automatically from video, and are combined with lexical features in a Bayesian generative model. The resulting multimodal system outperforms text-only seg- mentation on both manual and automatically- recognized speech transcripts. 1 Introduction When people communicate face-to-face, discourse cues are expressed simultaneously through multiple channels. Previous research has extensively studied how discourse cues correlate with lexico-syntactic and prosodic features (Hearst, 1994; Hirschberg and Nakatani, 1998; Passonneau and Litman, 1997); this work informs various text and speech processing applications, such as automatic summarization and segmentation. Gesture is another communicative modality that frequently accompanies speech, yet it has not been exploited for computational discourse analysis. This paper empirically demonstrates that gesture correlates with discourse structure. In particular, we show that automatically-extracted visual fea- tures can be combined with lexical cues in a sta- tistical model to predict topic segmentation, a fre- quently studied form of discourse structure. Our method builds on the idea that coherent discourse segments are characterized by gestural cohesion; in other words, that such segments exhibit homoge- neous gestural patterns. Lexical cohesion (Halliday and Hasan, 1976) forms the backbone of many ver- bal segmentation algorithms, on the theory that seg- mentation boundaries should be placed where the distribution of words changes (Hearst, 1994). With gestural cohesion, we explore whether the same idea holds for gesture features. The motivation for this approach comes from a series of psycholinguistic studies suggesting that gesture supplements speech with meaningful and unique semantic content (McNeill, 1992; Kendon, 2004). We assume that repeated patterns in gesture are indicative of the semantic coherence that charac- terizes well-defined discourse segments. An advan- tage of this view is that gestures can be brought to bear on discourse analysis without undertaking the daunting task of recognizing and interpreting indi- vidual gestures. This is crucial because coverbal gesture – unlike formal sign language – rarely fol- lows any predefined form or grammar, and may vary dramatically by speaker. A key implementational challenge is automati- cally extracting gestural information from raw video and representing it in a way that can applied to dis- course analysis. We employ a representation of vi- sual codewords, which capture clusters of low-level motion patterns. For example, one codeword may correspond to strong left-right motion in the up- per part of the frame. These codewords are then treated similarly to lexical items; our model iden- tifies changes in their distribution, and predicts topic 852 boundaries appropriately. The overall framework is implemented as …

Content truncated to 500 words