Referring to World Objects With Text and Pictures

REFERRING TO WORLD OBJECTS WITH TEXT AND PICTURES Elisabcth Andr6, Thomas Rist Gcrman Research Center for Artificial Intelligence (DFKI) D-66123 Saarbriickcn, Germany, c-mail: {andre, fist} @dfki.uni-sb.dc ABSTRACT: It oftcn makes sense to employ both text and pictures wheu referring to world objects. In this pa- per, we present a model for referring which is based on the assumption that concepts may be activated not only by Icxt, but also by pictures and tcxt-pieturc combiniltious. By means of a case study, we demouslrale that l'ailure aml suc- cess of referring acts can be cxplalncd by thc user's ability to infer ccrtaiu links between mental representations and object descriptions. Finally, we show how the model has been incorporated into a plan-t)ased multimedia prcseata- tion system by defiaiug operators lk)r concept activation. 1 INTRODUCTION From a speech act theoretical point of view, referring is a planncd action to achicve certalu go:ds (Appclt aud Kroa- fold, 1987). Although natur~ language may be the most conventional vehicle for referring, it has been widely ac- ccpted that pictures cau be used ~s well. For example, Goodmann (1969) points out that pictures can be cmploycd to refer to both an individual object and the type of which an objcct is an exemplary of. Morcovcr, there arc good reasons to include pictures in refcrring acts. l'icturcs ef- fectively convey discriminating object properties such as surface atlributes and shape. If au object can only be dis- criminated against alternatives through ils location, a pic- ture may provide the spatted context of the object. Since depictions arc explicit material representations of the world objects to which they correspond, new attributes of the type 'being dcpicted as ...' arc iutroducc(l which, in ttlrn, pro- vide an additiomd source for object discriminatiou (e.g., the knob which is reprcscnlcd by thc black circle ...). Last but not least, several graphical focusing tcchniqucs can bc applied to effcctivcly constraiu the set of alternatives (c.g., arrows, blinking). Unfortunately, there is also a dark side of the picture. An obvious drawback is that pictures do not provide for syutactical devices to distinguish between a reference-specifying and a predication-specifying part since objects and their properties are hardly separable once depict "cd. Auothcr difliculty is that pictures lack the means to distinguish deliuitc from indefinite descriptions. Thus, it may remain unclear whcthcr a particular object or whether ~m ,-u-biUzu-y exemplary of a class is depicted. The conclu- sion we can draw from these considerations is that it often makes sensc to employ bofll text lind pictures when rcfcr- riug to domain objects. Pictures may be used in order to simplify verbal reference expressions. On the other hand, ambiguitics of pictures cau be rcsolvcd by providing addi- tional information throngh text. When an~dyzing illustrated documeuls such as assembly matmals and iustructions for use, diffcrcnt kinds of rcfcrring expression can be found: Multimedia referring expressions rcfcr to world objects via a combination of at least two media. :Each medium con- vcys somc discriminatlug attributcs which in sum ,allow for a …

Content truncated to 500 words