Domain-Specific Lexical Grounding in Noisy Visual-Textual Documents

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2039–2045, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2039 Domain-Specific Lexical Grounding in Noisy Visual-Textual Documents Gregory Yauney Cornell University gyauney@cs.cornell.edu Jack Hessel Allen Institute for AI jackh@allenai.org David Mimno Cornell University mimno@cornell.edu Abstract Images can give us insights into the contex- tual meanings of words, but current image- text grounding approaches require detailed an- notations. Such granular annotation is rare, expensive, and unavailable in most domain- specific contexts. In contrast, unlabeled multi- image, multi-sentence documents are abun- dant. Can lexical grounding be learned from such documents, even though they have sig- nificant lexical and visual overlap? Work- ing with a case study dataset of real estate listings, we demonstrate the challenge of dis- tinguishing highly correlated grounded terms, such as “kitchen” and “bedroom”, and intro- duce metrics to assess this document similarity. We present a simple unsupervised clustering- based method that increases precision and re- call beyond object detection and image tag- ging baselines when evaluated on labeled sub- sets of the dataset. The proposed method is particularly effective for local contextual meanings of a word, for example associat- ing “granite” with countertops in the real es- tate dataset and with rocky landscapes in a Wikipedia dataset. 1 Introduction Multimodal data consisting of text and images is not only ubiquitous but increasingly diverse: libraries are digitizing visual-textual collections (British Library Labs, 2016; The Smithsonian, 2020); news organizations release over 1M images per year to accompany news articles (The Asso- ciated Press, 2020); and social media messages are rarely sent without visual accompaniment. In this work, we focus on one such specialized, multi- modal domain: New York City real estate listings from the website StreetEasy. To effectively index image-text datasets for search, retrieval, and other tasks, we need algo- rithms that learn connections between modalities, “granite” StreetEasy Wikipedia “chrysler” StreetEasy Wikipedia “architect” StreetEasy Wikipedia Figure 1: We identify domain-specific associations between words and images from unlabeled multi- sentence, multi-image documents. doing so from data that is naturally abundant. In documents that contain multiple images and sen- tences, there may be no explicit annotations for image-sentence associations or bounding box-word associations. As a result, existing image caption- ing/tagging methods are difficult to adapt to un- labeled multi-image, multi-sentence documents. Indeed, most prior image captioning work has fo- cused on rare and expensive single-image, single- caption collections such as MSCOCO, which fo- cuses on literal, context-free descriptions for 80 ob- ject types (Lin et al., 2014). Similarly, off-the-shelf object detectors may not account for contextual factors: to an ImageNet classifier, “pool” refers to a pool table (Russakovsky et al., 2015). In the specialized real estate context, “pool” commonly refers to a swimming pool. 2040 bananas hanging from a stall in a produce market a blue umbrella and a black backpack on the ground a bunch of birds that are on a big rock StreetEasy MSCOCO RQA DIY LonelyPlanet Story-SIS Story-DII The apartment features a private balcony, dark hardwood …

Content truncated to 500 words