Combining Deictic Gestures and Natural Language for Referent Identification

Combining Deictic Gestures and Natural Language for Referent Identification Alfred Kobsa, Jfirgen Allgayer, Carola Reddig, Norbert Reithinger Dagmar Schmauks, Karin Harbusch, Wolfgang Wahlster SFB 314: AI - Knowledge-Based Systems University of Saarbrficken D-6600 Saarbriicken 11 West Germany Abstract In virtually all current natural-language dialog systems, users can only refer to objects by using linguistic descriptions. How- ever, in human face-to-face conversation, participants fre= quently use various sorts of deictic gestures as well. In this paper, we will present the referent identification component of XTRA, a system for a natural-language access to expert sys- tems. XTRA allows the user to combine NL input together with pointing gestures on the terminal screen in order Io refer to objects on the display. Information about the location and type of this deietic gesture, as well as about the linguistic de- scription of the referred object, the case frame, and the dialog memory are utilized for identifying the object. The system is tolerant in respect to impreciseness of both the deictic and the natural language input. The user can thereby refer to objects more easily, avoid referential failures, and employ vague ev- eryday terms instead of precise technical notions. Keywords: Deixis, referent identification, NP analysis, parsing 1. Introduction Various aspects of referent identification by hearers have been investigated in the last few years: It has been studied as a pro- cess of noun phrase resolution and attribute comparison (Lip- kis 1982), as a planned action (Cohen 1981, 84), as a pro- cess which depends on focus (Grosz 1981), context (Reichman 1981), the mutual beliefs shared between speaker and hearer (Clark & Marshall 1.981) and the modality of linguistic com- munication (telephone vs. teletype, cf. Cohen 1984), and as a process which is prone to various sorts of conversational failure (Goodman 1985). In all of these studies, natural language is the only conversational medium. For identifying objects un- der discussion, the hearer can therefore only utilize the NL de- scriptions provided by the speaker, and information about the previous dialog and the task domain at hand. In face-to-face conversation, however, participants also fre- quently use extralinguistic means for referent identification, in particular, various sorts of deictic gestures (such as pointing at something by ones hand, finger, pencil, head or eyes). One This work is part of the SFB 314 Research Program on AI and Knowledge- Based Systems and has been supported by the German Science Founda- tion (DFG). The authors would like to thank J. Rink and W. Finkler for their help in preparing the manuscript, E-mail address of the authors: surname%sbsvamuucp~germany.csnet 356 may assume that this is done for simplifying and speeding up the identification process for both the hearer and the speaker, as well as avoiding referential failures. Certain technical in- novations in the last few years (e.g., high-resolution graphic displays, window systems, touch-sensitive screens, input via a pointing device such as the mouse or the light-pen) have made it possible for computational linguistics to also experiment with and study a certain class of …

Content truncated to 500 words