Incorporating Temporal and Semantic Information with Eye Gaze for Automatic Word Acquisition in Multimodal Conversational Systems

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 244–253, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Incorporating Temporal and Semantic Information with Eye Gaze for Automatic Word Acquisition in Multimodal Conversational Systems Shaolin Qu Joyce Y. Chai Department of Computer Science and Engineering Michigan State University East Lansing, MI 48824 {qushaoli,jchai}@cse.msu.edu Abstract One major bottleneck in conversational sys- tems is their incapability in interpreting un- expected user language inputs such as out-of- vocabulary words. To overcome this problem, conversational systems must be able to learn new words automatically during human ma- chine conversation. Motivated by psycholin- guistic findings on eye gaze and human lan- guage processing, we are developing tech- niques to incorporate human eye gaze for au- tomatic word acquisition in multimodal con- versational systems. This paper investigates the use of temporal alignment between speech and eye gaze and the use of domain knowl- edge in word acquisition. Our experiment re- sults indicate that eye gaze provides a poten- tial channel for automatically acquiring new words. The use of extra temporal and domain knowledge can significantly improve acquisi- tion performance. 1 Introduction Interpreting human language is a challenging prob- lem in human machine conversational systems due to the flexibility of human language behavior. When the encountered vocabulary is outside of the sys- tem’s knowledge, conversational systems tend to fail. It is desirable that conversational systems can learn new words automatically during human ma- chine conversation. While automatic word acquisi- tion in general is quite challenging, multimodal con- versational systems offer an unique opportunity to explore word acquisition. In a multimodal conversa- tional system where users can talk and interact with a graphical display, users’ eye gaze, which occurs naturally with speech production, provides a poten- tial channel for the system to learn new words auto- matically during human machine conversation. Psycholinguistic studies have shown that eye gaze is tightly linked to human language processing. Eye gaze is one of the reliable indicators of what a per- son is “thinking about” (Henderson and Ferreira, 2004). The direction of eye gaze carries informa- tion about the focus of the user’s attention (Just and Carpenter, 1976). The perceived visual context in- fluences spoken word recognition and mediates syn- tactic processing of spoken sentences (Tanenhaus et al., 1995). In addition, directly before speaking a word, the eyes move to the mentioned object (Grif- fin and Bock, 2000). Motivated by these psycholinguistic findings, we are investigating the use of eye gaze for automatic word acquisition in multimodal conversation. Par- ticulary, this paper investigates the use of tempo- ral information about speech and eye gaze and do- main semantic relatedness for automatic word ac- quisition. The domain semantic and temporal in- formation are incorporated in statistical translation models for word acquisition. Our experiments show that the use of domain semantic and temporal infor- mation significantly improves word acquisition per- formance. In the following sections, we first describe the ba- sic translation models for word acquisition. Then, we describe the …

Content truncated to 500 words