Who is “You”? Combining Linguistic and Gaze Features to Resolve Second-Person References in Dialogue

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 273–281, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Who is “You”? Combining Linguistic and Gaze Features to Resolve Second-Person References in Dialogue∗ Matthew Frampton1, Raquel Fern´andez1, Patrick Ehlen1, Mario Christoudias2, Trevor Darrell2 and Stanley Peters1 1Center for the Study of Language and Information, Stanford University {frampton, raquelfr, ehlen, peters}@stanford.edu 2International Computer Science Institute, University of California at Berkeley cmch@icsi.berkeley.edu, trevor@eecs.berkeley.edu Abstract We explore the problem of resolving the second person English pronoun you in multi-party dialogue, using a combination of linguistic and visual features. First, we distinguish generic and referential uses, then we classify the referential uses as ei- ther plural or singular, and finally, for the latter cases, we identify the addressee. In our first set of experiments, the linguistic and visual features are derived from man- ual transcriptions and annotations, but in the second set, they are generated through entirely automatic means. Results show that a multimodal system is often prefer- able to a unimodal one. 1 Introduction The English pronoun you is the second most fre- quent word in unrestricted conversation (after I and right before it).1 Despite this, with the ex- ception of Gupta et al. (2007b; 2007a), its re- solution has received very little attention in the lit- erature. This is perhaps not surprising since the vast amount of work on anaphora and reference resolution has focused on text or discourse - medi- ums where second-person deixis is perhaps not as prominent as it is in dialogue. For spoken di- alogue pronoun resolution modules however, re- solving you is an essential task that has an impor- tant impact on the capabilities of dialogue summa- rization systems. ∗We thank the anonymous EACL reviewers, and Surabhi Gupta, John Niekrasz and David Demirdjian for their com- ments and technical assistance. This work was supported by the CALO project (DARPA grant NBCH-D-03-0010). 1See e.g. http://www.kilgarriff.co.uk/BNC_lists/ Besides being important for computational im- plementations, resolving you is also an interesting and challenging research problem. As for third person pronouns such as it, some uses of you are not strictly referential. These include discourse marker uses such as you know in example (1), and generic uses like (2), where you does not refer to the addressee as it does in (3). (1) It’s not just, you know, noises like something hitting. (2) Often, you need to know specific button se- quences to get certain functionalities done. (3) I think it’s good. You’ve done a good review. However, unlike it, you is ambiguous between sin- gular and plural interpretations - an issue that is particularly problematic in multi-party conversa- tions. While you clearly has a plural referent in (4), in (3) the number of its referent is ambigu- ous.2 (4) I don’t know if you guys have any questions. When an utterance contains a singular referen- tial you, resolving the you amounts to identifying the individual to whom the utterance is addressed. This is …

Content truncated to 500 words