Automated Vocabulary Acquisition and Interpretation in Multimodal Conversational Systems

Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 368–375, Prague, Czech Republic, June 2007. c⃝2007 Association for Computational Linguistics Automated Vocabulary Acquisition and Interpretation in Multimodal Conversational Systems Yi Liu Joyce Y. Chai Rong Jin Department of Computer Science and Engineering Michigan State University East Lansing, MI 48824, USA {liuyi3, jchai, rongjin}@cse.msu.edu Abstract Motivated by psycholinguistic findings that eye gaze is tightly linked to human lan- guage production, we developed an unsuper- vised approach based on translation models to automatically learn the mappings between words and objects on a graphic display dur- ing human machine conversation. The ex- perimental results indicate that user eye gaze can provide useful information to establish such mappings, which have important impli- cations in automatically acquiring and inter- preting user vocabularies for conversational systems. 1 Introduction To facilitate effective human machine conversation, it is important for a conversational system to have knowledge about user vocabularies and understand how these vocabularies are mapped to the internal entities for which the system has representations. For example, in a multimodal conversational system that allows users to converse with a graphic inter- face, the system needs to know what vocabularies users tend to use to describe objects on the graphic display and what (type of) object(s) a user is attend- ing to when a particular word is expressed. Here, we use acquisition to refer to the process of acquir- ing relevant vocabularies describing internal entities, and interpretation to refer to the process of automat- ically identifying internal entities given a particular word. Both acquisition and interpretation have been traditionally approached by either knowledge engi- neering (e.g., manually created lexicons) or super- vised learning from annotated data. In this paper, we describe an unsupervised approach that relies on naturally co-occurred eye gaze and spoken utter- ances during human machine conversation to auto- matically acquire and interpret vocabularies. Motivated by psycholinguistic studies (Just and Carpenter, 1976; Griffin and Bock, 2000; Tenenhaus et al., 1995) and recent investigations on computa- tional models for language acquisition and ground- ing (Siskind, 1995; Roy and Pentland, 2002; Yu and Ballard, 2004), we are particularly interested in two unique questions related to multimodal conver- sational systems: (1) In a multimodal conversation that involves more complex tasks (e.g., both user initiated tasks and system initiated tasks), is there a reliable temporal alignment between eye gaze and spoken references so that the coupled inputs can be used for automated vocabulary acquisition and inter- pretation? (2) If such an alignment exists, how can we model this alignment and automatically acquire and interpret the vocabularies? To address the first question, we conducted an empirical study to examine the temporal relation- ships between eye fixations and their correspond- ing spoken references. As shown later in section 4, although a larger variance (compared to the find- ings from psycholinguistic studies) exists in terms of how eye gaze is linked to speech production during human machine conversation, eye fixations and the corresponding spoken references still occur in a …

Content truncated to 500 words