Fusing Eye Gaze with Speech Recognition Hypotheses to Resolve Exophoric References in Situated Dialogue

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 471–481, MIT, Massachusetts, USA, 9-11 October 2010. c⃝2010 Association for Computational Linguistics Fusing Eye Gaze with Speech Recognition Hypotheses to Resolve Exophoric References in Situated Dialogue Zahar Prasov and Joyce Y. Chai Department of Computer Science and Engineering Michigan State University East Lansing, MI 48824, USA {prasovza,jchai}@cse.msu.edu Abstract In situated dialogue humans often utter lin- guistic expressions that refer to extralinguistic entities in the environment. Correctly resolv- ing these references is critical yet challeng- ing for artificial agents partly due to their lim- ited speech recognition and language under- standing capabilities. Motivated by psycholin- guistic studies demonstrating a tight link be- tween language production and human eye gaze, we have developed approaches that in- tegrate naturally occurring human eye gaze with speech recognition hypotheses to resolve exophoric references in situated dialogue in a virtual world. In addition to incorporat- ing eye gaze with the best recognized spo- ken hypothesis, we developed an algorithm to also handle multiple hypotheses modeled as word confusion networks. Our empirical re- sults demonstrate that incorporating eye gaze with recognition hypotheses consistently out- performs the results obtained from processing recognition hypotheses alone. Incorporating eye gaze with word confusion networks fur- ther improves performance. 1 Introduction Given a rapid growth in virtual world applications for tutoring and training, video games and simu- lations, and assistive technology, enabling situated dialogue in virtual worlds has become increasingly important. Situated dialogue allows human users to navigate in a spatially rich environment and carry a conversation with artificial agents to achieve spe- cific tasks pertinent to the environment. Different from traditional telephony-based spoken dialogue systems and multimodal conversational interfaces, situated dialogue supports immersion and mobility in a visually rich environment and encourages so- cial and collaborative language use (Byron et al., 2005; Gorniak et al., 2006). In situated dialogue, hu- man users often need to make linguistic references, known as exophoric referring expressions (e.g., the book to the right), to extralinguistic entities in the environment. Reliably resolving these ref- erences is critical for dialogue success. However, reference resolution remains a challenging problem, partly due to limited speech and language process- ing capabilities caused by poor speech recognition (ASR), ambiguous language, and insufficient prag- matic knowledge. To address this problem, motivated by psycholin- guistic studies demonstrating a close relationship between language production and eye gaze, our previous work has incorporated naturally occurring eye gaze in reference resolution (Prasov and Chai, 2008). Our findings have shown that eye gaze can partially compensate for limited language process- ing and domain modeling. However, this work was conducted in a setting where users only spoke to a static visual interface. In situated dialogue, human speech and eye gaze patterns are much more com- plex. The dynamic nature of the environment and the complexity of spatially rich tasks have a massive influence on what the user will look at and say. It is not clear to what degree prior findings can generalize to situated …

Content truncated to 500 words