Exploring and Exploiting the Limited Utility of Captions in Recognizing Intention in Information Graphics

Proceedings of the 43rd Annual Meeting of the ACL, pages 223–230, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics Exploring and Exploiting the Limited Utility of Captions in Recognizing Intention in Information Graphics∗ Stephanie Elzer1 and Sandra Carberry2 and Daniel Chester2 and Seniz Demir2 and Nancy Green3 and Ingrid Zukerman4 and Keith Trnka2 1Dept. of Computer Science, Millersville University, Millersville, PA 17551 2Dept. of Computer Science, University of Delaware, Newark, DE 19716 3Dept. of Mathematical Sciences, Univ. of NC at Greensboro, Greensboro, NC 27402 4School of CS & Software Engrg, Monash Univ., Clayton, Victoria 3800 Australia Abstract This paper presents a corpus study that ex- plores the extent to which captions con- tribute to recognizing the intended mes- sage of an information graphic. It then presents an implemented graphic interpre- tation system that takes into account a va- riety of communicative signals, and an evaluation study showing that evidence obtained from shallow processing of the graphic’s caption has a significant impact on the system’s success. This work is part of a larger project whose goal is to provide sight-impaired users with effective access to information graphics. 1 Introduction Language research has posited that a speaker or writer executes a speech act whose intended mean- ing he expects the listener to be able to deduce, and that the listener identifies the intended meaning by reasoning about the observed signals and the mutual beliefs of author and interpreter (Grice, 1969; Clark, 1996). But as noted by Clark (Clark, 1996), lan- guage is more than just words. It is any “signal” (or lack of signal when one is expected), where a sig- nal is a deliberate action that is intended to convey a message. Although some information graphics are only in- tended to display data values, the overwhelming ma- jority of the graphics that we have examined (taken ∗Authors can be reached via email as fol- lows: elzer@cs.millersville.edu, nlgreen@uncg.edu, {carberry, chester, demir, trnka}@cis.udel.edu, In- grid.Zukerman@infotech.monash.edu.au. 1998 1999 2000 2001 1000 1500 2000 2500 3000 personal filings Local bankruptcy Figure 1: Graphic from a 2001 Local Newspaper from newspaper, magazine, and web articles) ap- pear to have some underlying goal or intended mes- sage, such as the graphic in Figure 1 whose com- municative goal is ostensibly to convey the sharp in- crease in local bankruptcies in the current year com- pared with the previous decreasing trend. Applying Clark’s view of language, it is reasonable to presume that the author of an information graphic expects the viewer to deduce from the graphic the message that the graphic was intended to convey, by reasoning about the graphic itself, the salience of entities in the graphic, and the graphic’s caption. This paper adopts Clark’s view of language as any deliberate signal that is intended to convey a mes- sage. Section 3 investigates the kinds of signals used in information graphics. Section 4 presents a cor- pus study that investigates the extent to which cap- tions capture the message of the graphic, illustrates the issues that …

Content truncated to 500 words