Natural and Simulated Pointing

NATURAL AND SIMULATED POINTING Dagmar Schmauks Sonderforschungsbereich 314 FB 10.2 - Informatik Universi6it des Saarlandes D-6600 Saarbrficken 11 WEST GERMANY CSnet: schmauks % sbsvax.uucp@ germany.csnet Abstract Referent identification in human conversation is performed both by describing the objects in question and by pointing at them. Up till now, only the linguis- tic component could be simulated in dialog systems. But recently, technical innovations have made it possible to 'point' at the objects on a display as well. The paper has two intentions. First, it investigates nat- ural pointing in more detail and offers some possibilities to classify the great variety of pointing actions. Then, it tries to clarify the extent to which pointing by techni- cal means (especially mouse-clicks) can be regarded as a simulation of natural pointing or as a functional equiv- alent. Furthermore, some steps towards even more ac- curate simulation are briefly mentioned. 1. Introduction 1.1 Terminological remarks The term 'deixis' denotes those referential devices whose interpretation requires a consideration of the sit- uation of utterance. "Local deixis' means the specifi- cation of directions, places and objects relative to the speaker's actual orientation. The closely related top- ics 'anaphora' and 'text-deixis' are not treated in this paper. One component of deictic actions are linguis- tic expressions, mainly demonstrative pronouns ('this', 'that') and adverbs ('here', 'there'), the other being ex- tralinguistic means, especially pointing gestures. In this paper, the latter are represented by the sign '/" ' "Natural pointing' denotes pointing actions occur- ring during interhuman dialog. This includes the use of easily available aids like pencils or pointers. "Simulated pohuing' means the use of technical pointing devices during man-computer dialog. Following the terminology of Clark, Schreuder, and Buttrick (1983), the object pointed at is called the 'demonstratum', and the descriptive part of the accom- panying noun phrase (if there is one) is called the 'de- scriptor'. The 'referent" is the object to which the whole pointing act is intended to refer. 1.2 Motivation In face-to-face interaction, pointing gestures are used frequently and efficiently. Although their referen- tial power is beyond any doubt, they have, up till now, hardly ever been treated in more detail. The disciplines concerned with them are mainly semiotics, linguistics and psychology. Recently, the investigation of natural pointing has also become interesting for the area of Artificial Intel- ligence. In dialog systems developed to date, objects could be referred to by more or less complex verbal de- scriptions or unique artificial identifiers only. Techni- cal innovations (e.g., high-resolution graphic displays, touch-sensitive screens, pointing devices such as mice or joysticks) have made it possible to simulate pointing gestures to various degrees as well. Multimodal input is both more comfortable from the user's point of view, and a more natural simulation of interhuman commu- nication. Therefore, several systems have been developed re- cently which allow the combination of verbal descrip- tions and pointing gestures for referent identification (see section 5.2). One example is the dialog system XTRA (a natural-language access system for expert systems) which is …

Content truncated to 500 words