Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets

Proceedings of NAACL-HLT 2018, pages 2194–2205 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Quantifying the visual concreteness of words and topics in multimodal datasets Jack Hessel Cornell University jhessel@cs.cornell.edu David Mimno Cornell University mimno@cornell.edu Lillian Lee Cornell University llee@cs.cornell.edu Abstract Multimodal machine learning algorithms aim to learn visual-textual correspondences. Previous work suggests that concepts with concrete visual manifestations may be eas- ier to learn than concepts with abstract ones. We give an algorithm for automatically computing the visual concreteness of words and topics within multimodal datasets. We apply the approach in four settings, ranging from image captions to images/text scraped from historical books. In addition to en- abling explorations of concepts in multi- modal datasets, our concreteness scores predict the capacity of machine learning algorithms to learn textual/visual relation- ships. We find that 1) concrete concepts are indeed easier to learn; 2) the large num- ber of algorithms we consider have similar failure cases; 3) the precise positive rela- tionship between concreteness and perfor- mance varies between datasets. We con- clude with recommendations for using con- creteness scores to facilitate future multi- modal research. 1 Introduction Text and images are often used together to serve as a richer form of content. For example, news articles may be accompanied by photographs or infographics; images shared on social media are often coupled with descriptions or tags; and text- books include illustrations, photos, and other vi- sual elements. The ubiquity and diversity of such “text+image” material (henceforth referred to as multimodal content) suggest that, from the stand- point of sharing information, images and text are often natural complements. Ideally, machine learning algorithms that in- corporate information from both text and images should have a fuller perspective than those that con- sider either text or images in isolation. But Hill and Korhonen (2014b) observe that for their particular multimodal architecture, the level of concreteness of a concept being represented — intuitively, the idea of a dog is more concrete than that of beauty — affects whether multimodal or single-channel representations are more effective. In their case, concreteness was derived for 766 nouns and verbs from a fixed psycholinguistic database of human ratings. In contrast, we introduce an adaptive algorithm for characterizing the visual concreteness of all the concepts indexed textually (e.g., “dog”) in a given multimodal dataset. Our approach is to leverage the geometry of image/text space. Intuitively, a visually concrete concept is one associated with more locally similar sets of images; for example, images associated with “dog” will likely contain dogs, whereas images associated with “beautiful” may contain flowers, sunsets, weddings, or an abun- dance of other possibilities — see Fig. 1. Allowing concreteness to be dataset-specific is an important innovation because concreteness is contextual. For example, in one dataset we work with, our method scores “London” as highly con- crete because of a preponderance of iconic London images in it, such as Big Ben and double-decker buses; whereas for a separate dataset, “London” is used as …

Content truncated to 500 words