Conditional Modality Fusion for Coreference Resolution

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 352–359, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Conditional Modality Fusion for Coreference Resolution Jacob Eisenstein and Randall Davis Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, MA 02139 USA {jacobe,davis}@csail.mit.edu Abstract Non-verbal modalities such as gesture can improve processing of spontaneous spoken language. For example, similar hand ges- tures tend to predict semantic similarity, so features that quantify gestural similarity can improve semantic tasks such as coreference resolution. However, not all hand move- ments are informative gestures; psycholog- ical research has shown that speakers are more likely to gesture meaningfully when their speech is ambiguous. Ideally, one would attend to gesture only in such cir- cumstances, and ignore other hand move- ments. We present conditional modality fusion, which formalizes this intuition by treating the informativeness of gesture as a hidden variable to be learned jointly with the class label. Applied to coreference resolution, conditional modality fusion sig- nificantly outperforms both early and late modality fusion, which are current tech- niques for modality combination. 1 Introduction Non-verbal modalities such as gesture and prosody can increase the robustness of NLP systems to the inevitable disfluency of spontaneous speech. For ex- ample, consider the following excerpt from a dia- logue in which the speaker describes a mechanical device: “So this moves up, and it – everything moves up. And this top one clears this area here, and goes all the way up to the top.” The references in this passage are difficult to disambiguate, but the gestures shown in Figure 1 make the meaning more clear. However, non-verbal modalities are often noisy, and their interactions with speech are complex (McNeill, 1992). Ges- ture, for example, is sometimes communicative, but other times merely distracting. While people have little difficulty distinguishing between meaningful gestures and irrelevant hand motions (e.g., self- touching, adjusting glasses) (Goodwin and Good- win, 1986), NLP systems may be confused by such seemingly random movements. Our goal is to in- clude non-verbal features only in the specific cases when they are helpful and necessary. We present a model that learns in an unsupervised fashion when non-verbal features are useful, allow- ing it to gate the contribution of those features. The relevance of the non-verbal features is treated as a hidden variable, which is learned jointly with the class label in a conditional model. We demonstrate that this improves performance on binary corefer- ence resolution, the task of determining whether a noun phrases refers to a single semantic entity. Con- ditional modality fusion yields a relative increase of 73% in the contribution of hand-gesture features. The model is not specifically tailored to gesture- speech integration, and may also be applicable to other non-verbal modalities. 2 Related work Most of the existing work on integrating non-verbal features relates to prosody. For example, Shriberg et al. (2000) explore the use of prosodic features for sentence and topic segmentation. The first modal- 352 And this top …

Content truncated to 500 words