Detecting Visual Text

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 762–772, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Detecting Visual Text Jesse Dodge1, Amit Goyal2, Xufeng Han3, Alyssa Mensch4, Margaret Mitchell5, Karl Stratos6 Kota Yamaguchi3, Yejin Choi3, Hal Daum´e III2, Alexander C. Berg3 and Tamara L. Berg3 1University of Washington, 2University of Maryland, 3Stony Brook University 4MIT, 5Oregon Health & Science University, 6Columbia University dodgejesse@gmail.com, amit@umiacs.umd.edu, xufhan@cs.stonybrook.edu acmensch@mit.edu, mitchmar@ohsu.edu, stratos@cs.columbia.edu kyamagu@cs.stonybrook.edu, ychoi@cs.stonybrook.edu me@hal3.name, aberg@cs.stonybrook.edu, tlberg@cs.stonybrook.edu Abstract When people describe a scene, they often in- clude information that is not visually apparent; sometimes based on background knowledge, sometimes to tell a story. We aim to sepa- rate visual text—descriptions of what is being seen—from non-visual text in natural images and their descriptions. To do so, we first con- cretely define what it means to be visual, an- notate visual text and then develop algorithms to automatically classify noun phrases as vi- sual or non-visual. We find that using text alone, we are able to achieve high accuracies at this task, and that incorporating features derived from computer vision algorithms im- proves performance. Finally, we show that we can reliably mine visual nouns and adjectives from large corpora and that we can use these effectively in the classification task. 1 Introduction People use language to describe the visual world. Our goal is to: formalize what “visual text” is (Sec- tion 2.2); analyze naturally occurring written lan- guage for occurrences of visual text (Section 2); and build models that can detect visual descriptions from raw text or from image/text pairs (Section 3). This is a challenging problem. One challenge is demon- strated in Figure 1, which contains two images that contain the noun “car” in their human-written cap- tions. In one case (the top image), there actually is a car in the image; in the other case, there is not: the car refers to the state of the speaker. The ability to automatically identify visual text is practically useful in a number of scenarios. One can Another dream car to add to the list, this one spotted in Hanbury St. Shot out my car win- dow while stuck in traf- fic because people in Cincinnati can’t drive in the rain. Figure 1: Two image/caption pairs, both containing the noun “car” but only the top one in a visual context. imagine automatically mining image/caption data (like that in Figure 1) to train object recognition sys- tems. However, in order to do so reliably, one must know whether the “car” actually appears or not. When building image search engines, it is common to use text near an image as features; this is more useful when this text is actually visual. Or when training systems to automatically generate captions of images (e.g., for visually impaired users), we need good language models for visual text. One of our goals is to define what it means for a bit of text to be visual. As inspiration, we consider …

Content truncated to 500 words