Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4622–4632, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4622 Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts Julia Kruk1∗, Jonah Lubin2∗, Karan Sikka1, Xiao Lin1, Dan Jurafsky3, Ajay Divakaran1†, 1SRI International, Princeton, NJ∗equal contribution 2The University of Chicago, Chicago, Illinois 3Stanford University, Stanford, CA Abstract Computing author intent from multimodal data like Instagram posts requires modeling a complex relationship between text and image. For example, a caption might evoke an ironic contrast with the image, so neither caption nor image is a mere transcript of the other. In- stead they combine—via what has been called meaning multiplication Bateman (2014)—to create a new meaning that has a more complex relation to the literal meanings of text and im- age. Here we introduce a multimodal dataset of 1299 Instagram posts labeled for three or- thogonal taxonomies: the authorial intent be- hind the image-caption pair, the contextual re- lationship between the literal meanings of the image and caption, and the semiotic relation- ship between the signified meanings of the im- age and caption. We build a baseline deep multimodal classifier to validate the taxonomy, showing that employing both text and image improves intent detection by 9.6% compared to using only the image modality, demonstrat- ing the commonality of non-intersective mean- ing multiplication. The gain with multimodal- ity is greatest when the image and caption di- verge semiotically. Our dataset offers a new resource for the study of the rich meanings that result from pairing text and image. 1 Introduction Multimodal social platforms such as Instagram let content creators combine visual and textual modalities. The resulting widespread use of text+image makes interpreting author intent in multimodal messages an important task for NLP for document understanding. There are many recent language processing studies of images accompanied by basic text labels ∗ Work done while Julia (from Cornell University) and Jonah were interns at SRI International. † Corresponding author, ajay.divakaran@sri.com. Figure 1: Image-Caption meaning multiplication: A change in the caption completely changes the overall meaning of the image-caption pair. or captions (Chen et al., 2015; Faghri et al., 2018, inter alia). But prior work on image–text pairs has generally been asymmetric, regarding either im- age or text as the primary content, and the other as mere complement. Scholars from semiotics as well as computer science have pointed out that this is insufficient; often text and image are not com- bined by a simple addition or intersection of the component meanings (Bateman, 2014; Marsh and Domas White, 2003; Zhang et al., 2018). Rather, determining author intent with text+image content requires a richer kind of meaning composition that has been called mean- ing multiplication (Bateman, 2014): the creation of new meaning through integrating image and text. Meaning multiplication includes simple meaning intersection or concatenation (a picture of a dog with the label “dog”, or the label “Ru- fus”). …

Content truncated to 500 words