Categorizing and Inferring the Relationship between the Text and Image of Twitter Posts

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2830–2840 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2830 Categorizing and Inferring the Relationship between the Text and Image of Twitter Posts Alakananda Vempala Bloomberg LP avempala@bloomberg.net Daniel Preot¸iuc-Pietro Bloomberg LP dpreotiucpie@bloomberg.net Abstract Text in social media posts is frequently accom- panied by images in order to provide content, supply context, or to express feelings. This pa- per studies how the meaning of the entire tweet is composed through the relationship between its textual content and its image. We build and release a data set of image tweets anno- tated with four classes which express whether the text or the image provides additional in- formation to the other modality. We show that by combining the text and image infor- mation, we can build a machine learning ap- proach that accurately distinguishes between the relationship types. Further, we derive in- sights into how these relationships are mate- rialized through text and image content anal- ysis and how they are impacted by user de- mographic traits. These methods can be used in several downstream applications including pre-training image tagging models, collecting distantly supervised data for image captioning, and can be directly used in end-user applica- tions to optimize screen estate. 1 Introduction Social media sites have traditionally been cen- tered around publishing textual content. Recently, posting images on social media has become a very popular way of expressing content and feel- ings especially due to the wide availability of mo- bile devices and connectivity. Images are cur- rently present in a significant fraction of tweets and tweets with images get double the engagement of those without (Buffer, 2016). Thus, in addition to text, images have become key components of tweets. However, little is known about how textual con- tent is related to the images with which they ap- pear. For example, concepts or feelings mentioned in text could be illustrated or strengthened by im- ages, text can point to the content of an image or This is what happens when you lock your bike to a sign Awesome! (a) Image adds to the tweet meaning & Text is represented in image (b) Image adds to the tweet meaning & Text is not repre- sented in image Tacos are the best Last exam turned in. No more juggling work + school + family + hobbies. Maybe now they’ll fi- nally give me a BSc (c) Image does not add to mean- ing & Text is represented in im- age (d) Image does not add to mean- ing & Text is not represented in image Figure 1: Examples of the four types of text-image re- lationship from this study. can just provide commentary on the image con- tent. Formalizing and understanding the relation- ship between the two modalities – text and images – is useful in several areas: a) for NLP and computer vision research, where image and text data from tweets are used to …

Content truncated to 500 words