Co-reference via Pointing and Haptics in Multi-Modal Dialogues

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 523–527, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Co-reference via Pointing and Haptics in Multi-Modal Dialogues Lin Chen, Barbara Di Eugenio Department of Computer Science University of Illinois at Chicago 851 S Morgan ST, Chicago, IL 60607, USA {lchen43,bdieugen}@uic.edu Abstract This paper describes our ongoing work on resolving third person pronouns and deictic words in a multi-modal corpus. We show that about two thirds of these referring expressions have antecedents that are introduced by point- ing gestures or by haptic-ostensive actions (actions that involve manipulating an object). After describing our annotation scheme, we discuss the co-reference models we learn from multi-modal features. The usage of haptic- ostensive actions in a co-reference model is a novel contribution of our work. 1 Introduction Co-reference resolution has received a lot of atten- tion. However, as Eisenstein and Davis (2006) noted, most research on co-reference resolution has focused on written text. This task is much more difficult in dialogue, especially in multi-modal di- alogue contexts. First, utterances are informal, un- grammatical and disfluent. Second, people sponta- neously use gestures and other body language. As noticed by Kehler (2000), Goldin-Meadow (2003), and Chen et al. (2011), in a multi-modal corpus, the antecedents of referring expressions are often in- troduced via gestures. Whereas the role played by pointing gestures in referring has been studied, the same is not true for other types of gestures. In this paper, alongside pointing gestures, we will discuss the role played by Haptic-Ostensive (H-O) actions, i.e., referring to an object by manipulating it in the world (Landragin et al., 2002; Foster et al., 2008). As far as we know, no computational models of co- reference have been developed that include H-O ac- tions: (Landragin et al., 2002) focused on percep- tual salience and (Foster et al., 2008) on generation rather than interpretation. We should point out that at the time of writing we only focus on resolving third person pronouns and deictics. The rest of this paper is organized as follows. In Section 2 we describe our multi-modal annotation scheme. In Section 3 we present the pronoun/deictic resolution system. In Section 4, we discuss experi- ments and results. 2 The Data Set The dataset we use in this paper is a subset of the ELDERLY-AT-HOME corpus (Di Eugenio et al., 2010), a multi-modal corpus in the domain of elderly care. It contains 20 human-human dialogues. In each dialogue, a helper (HEL) and an elderly person (ELD) performed Activities of Daily Living (Krapp, 2002), such as getting up from chairs, finding pots, cooking pastas, in a realistic setting, a studio apart- ment used for teaching and research. The corpus contains videos and voice data in avi format, haptics data collected via instrumented gloves in csv format, and the transcribed utterances in xml format. We focused on specific subdialogues in this cor- pus, that we call Find tasks: a Find task is a …

Content truncated to 500 words