Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements

Proceedings of the 28th International Conference on Computational Linguistics, pages 1886–1892 Barcelona, Spain (Online), December 8-13, 2020 1886 Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements Andrey Savchenko1,3, Anton Alekseev1, Sejeong Kwon2, Elena Tutubalina1, Evgeniy Miasnikov1 and Sergey Nikolenko1,4 1Samsung-PDMI Joint AI Center, Steklov Institute of Mathematics at St. Petersburg / St. Petersburg, Russia 2Samsung Research / Seoul, Korea 3National Research University Higher School of Economics / Nizhny Novgorod, Russia 4Neuromation OU / Tallinn, Estonia andrey.v.savchenko@gmail.com, anton.m.alexeyev@gmail.com, sejeong.kwon@samsung.com, tlenusik@gmail.com, deeprehension@gmail.com, sergey@logic.pdmi.ras.ru Abstract Understanding image advertisements is a challenging task, often requiring non-literal interpreta- tion. We argue that standard image-based predictions are insufficient for symbolism prediction. Following the intuition that texts and images are complementary in advertising, we introduce a multimodal ensemble of a state of the art image-based classifier, a classifier based on an ob- ject detection architecture, and a fine-tuned language model applied to texts extracted from ads by OCR. The resulting system establishes a new state of the art in symbolism prediction. 1 Introduction Visual advertisement, both image and video, can efficiently convey persuasive messages to potential customers. Much of the power of visual advertising comes from multiple ways to interact with the user, embedding messages in both literal and symbolic forms. One of the most complex tasks in ad analysis is symbol interpretation, a much harder problem than, for instance, object detection. For example, in the LEE denim jeans brand ad shown in Fig. 1 a human figure is depicted with a lion’s head, which might arguably symbolize “courage”, “strong character”, etc. In real life, lions are rarely seen wearing denim jeans, so analyzing this scene with a machine learning model trained on regular photographs would be quite challenging at least in terms of object detection. Moreover, establishing a direct association between the (possibly detected) lion and human “courage” is a hard problem by itself. Both factors make symbol interpretation difficult. But understanding ads computationally has important applications: decoded messages can help improve ad targeting, tools for message understanding can lead to better descriptions of visual content, and they can also inform the users how exactly they are being persuaded. Hussain et al. (2017) present a crowdsourced dataset of advertisements, including images and videos. They introduced several annotation tasks: topic detection, sentiment detection, symbol recognition, strat- egy analysis, slogan annotation, and question answering for texts related to the ads’ messages and mo- tivation. In this work we focus on the image-based multi-label classification task for symbols. In this problem, each annotated image in the dataset has several bounding boxes and textual description that can be mapped to a limited number of categories of symbols. Related work has mostly concentrated on this dataset, and it often combines text and images, but problem settings vary widely. In particular, the AD- VISE framework embeds text and images in a joint embedding space in the context of choosing human statements describing the ad’s message (Ye and Kovashka, 2017). Zhang et al. (2018) introduce various feature extraction methods to capture …

Content truncated to 500 words