An Exploration of Eye Gaze in Spoken Language Processing for Multimodal Conversational Interfaces

Proceedings of NAACL HLT 2007, pages 284–291, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics An Exploration of Eye Gaze in Spoken Language Processing for Multimodal Conversational Interfaces Shaolin Qu Joyce Y. Chai Department of Computer Science and Engineering Michigan State University East Lansing, MI 48824 {qushaoli,jchai}@cse.msu.edu Abstract Motivated by psycholinguistic findings, we are currently investigating the role of eye gaze in spoken language understand- ing for multimodal conversational sys- tems. Our assumption is that, during hu- man machine conversation, a user’s eye gaze on the graphical display indicates salient entities on which the user’s atten- tion is focused. The specific domain infor- mation about the salient entities is likely to be the content of communication and therefore can be used to constrain speech hypotheses and help language understand- ing. Based on this assumption, this paper describes an exploratory study that incor- porates eye gaze in salience modeling for spoken language processing. Our empiri- cal results show that eye gaze has a poten- tial in improving automated language pro- cessing. Eye gaze is subconscious and in- voluntary during human machine conver- sation. Our work motivates more in-depth investigation on eye gaze in attention pre- diction and its implication in automated language processing. 1 Introduction Psycholinguistic experiments have shown that eye gaze is tightly linked to human language process- ing. Eye gaze is one of the reliable indicators of what a person is “thinking about” (Henderson and Ferreira, 2004). The direction of gaze carries infor- mation about the focus of the users attention (Just and Carpenter, 1976). The perceived visual context influences spoken word recognition and mediates syntactic processing (Tanenhaus et al., 1995; Roy and Mukherjee, 2005). In addition, directly before speaking a word, the eyes move to the mentioned object (Griffin and Bock, 2000). Motivated by these psycholinguistic findings, we are currently investigating the role of eye gaze in spoken language understanding during human ma- chine conversation. Through multimodal interfaces, a user can look at a graphic display and converse with the system at the same time. Our assumption is that, during human machine conversation, a user’s eye gaze on the graphical display can indicate salient entities on which the user’s attention is focused. The specific domain information about the salient enti- ties is likely linked to the content of communication and therefore can be used to constrain speech hy- potheses and influence language understanding. Based on this assumption, we carried out an ex- ploration study where eye gaze information is in- corporated in a salience model to tailor a language model for spoken language processing. Our prelim- inary results show that eye gaze can be useful in im- proving spoken language processing and the effect of eye gaze varies among different users. Because eye gaze is subconscious and involuntary in human machine conversation, our work also motivates sys- tematic investigations on how eye gaze contributes to attention prediction and its implications in auto- mated language processing. 2 Related Work Eye gaze has been mainly used in human machine …

Content truncated to 500 words