Multi-modal Reference Resolution in Situated Dialogue by Integrating Linguistic and Extra-Linguistic Clues

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 84–92, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Multi-modal Reference Resolution in Situated Dialogue by Integrating Linguistic and Extra-Linguistic Clues Ryu Iida, Masaaki Yasuhara, Takenobu Tokunaga Department of Computer Science, Tokyo Institute of Technology W8-73, 2-12-1 Ohokayama Meguro Tokyo, 152-8552 Japan {ryu-i,yasuhara,take}@cl.cs.titech.ac.jp Abstract This paper focuses on examining the effect of extra-linguistic information, such as eye gaze, integrated with linguistic informa- tion on multi-modal reference resolution. In our evaluation, we employ eye gaze information together with other linguistic factors in machine learning, while in prior work such as Kelleher (2006) and Prasov and Chai (2008) the incorporation of eye gaze and linguistic clues was heuristically realised. Conducting our empirical evalu- ation using a data set extended the REX-J corpus (Spanger et al., 2010) including eye gaze information, we examine which types of clues are useful on these three data sets, which consist largely of pronouns, non- pronouns and both respectively. Our re- sults demonstrate that a dynamically mov- ing visible indicator within the computer display (e.g. a mouse cursor) contributes to reference resolution for pronouns, while eye gaze information is more useful for the resolution of non-pronouns. 1 Introduction The task of reference resolution has received much attention because it is important for applications that require interpreting text. In recent work on reference resolution within a text, several ma- chine learning-based approaches have been pro- posed (McCarthy and Lehnert, 1995; Ge et al., 1998; Soon et al., 2001; Ng and Cardie, 2002; Iida et al., 2003; Yang et al., 2003; Denis and Baldridge, 2008), each of which mainly exploits linguistic clues motivated by the Centering The- ory (Grosz et al., 1995) to model the discourse salience of all candidate antecedents. For instance, Yang et al. (2003) and Iida et al. (2003) presented machine learning-based reference resolution mod- els where a pairwise comparison of candidate an- tecedents, in line with the basic idea of the Cen- tering Theory, leads to the selection of the candi- date with the highest salience for a given context. Denis and Baldridge (2008) extended the model by integrating the set of pairwise comparisons into ranking candidates to directly learn which clues of antecedents are useful. Through the empirical evaluations using the data sets provided by the Message Understand- ing Conference (MUC)1 and the Automatic Con- tent Extraction (ACE)2, which consist of newspa- per articles and transcripts of broadcasts, linguis- tically motivated approaches have achieved better performance than state-of-the-art rule-based refer- ence resolution systems (e.g. Soon et al. (2001) and Ng and Cardie (2002)). In contrast to this research paradigm (i.e. re- search focusing on only the linguistic aspect of reference), research in the area of multi-modal interfaces has focused on referring expressions used in multi-modal conversations, in other words, identifying referents of referring expressions in a static scene or a situated world (e.g. objects depicted in a computer display), taking extra- linguistic clues into account (Byron, 2005; Prasov and Chai, 2008; …

Content truncated to 500 words