Understanding Advertisements with BERT

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7542–7547 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7542 Understanding Advertisements with BERT Kanika Kalra, Bhargav Kurma, Silpa Sreelatha, Manasi Patwardhan, Shirish Karande TCS Research, Pune, India {kalra.kanika, bhargav.kurma, silpa.sreelatha, manasi.patwardhan, shirish.karande} @tcs.com Abstract We consider a task based on CVPR 2018 challenge dataset on advertisement (Ad) un- derstanding. The task involves detecting the viewer’s interpretation of an Ad image cap- tured as text. Recent results have shown that the embedded scene-text in the image holds a vital cue for this task. Motivated by this, we fine-tune the base BERT model for a sentence- pair classification task. Despite utilizing the scene-text as the only source of visual informa- tion, we could achieve a hit-or-miss accuracy of 84.95% on the challenge test data. To en- able BERT to process other visual information, we append image captions to the scene-text. This achieves an accuracy of 89.69%, which is an improvement of 4.7%. This is the best reported result for this task. 1 Introduction The advertisement understanding challenge dataset of CVPR 2018 collected textual inputs from a set of viewers to capture their interpretations of Ad im- ages (Hussain et al., 2017). The task is to rank the given valid and negatively sampled invalid interpre- tations of an image. Initial approaches to the prob- lem tried capturing the visual semantics with a com- bination of object proposal features and relation- ships of objects with common symbolism (Doshi and Hinthorn, 2018; Ye and Kovashka, 2018; Ahuja et al., 2018). Recently, Dey et al. (2019a,b) have obtained a significant improvement in performance by utilizing the text embedded in the image (termed as the scene-text) as another channel of informa- tion. These approaches do not evaluate the validity of an interpretation by using attention to associate the words and phrases of the interpretation to frag- ments of textual and visual cues in the image. For example, in the Ad of the car company in Figure 1, the words and phrases from a viewer’s input ‘I should buy this car because it would add some Figure 1: Ads Dataset: Textual and Visual Cues excitement to my life’ can be associated with the object ‘car’ in the image and the phrase ‘add spark to life’ in the scene-text. To capture these map- pings, we need a model that can simultaneously pay attention to the image and the interpretations at various levels of granularity. The recently proposed BERT pre-trained lan- guage model (Devlin et al., 2019) has provided excellent performance on several NLP tasks. The underlying attention-based transformer architec- ture (Vaswani et al., 2017) allows BERT to capture contextual representations. We leverage the pre- trained base BERT model to capture a contextual representation of the viewer’s interpretation with respect to the visual and textual cues in the image. One of the challenges we face is to provide information on visual cues to the BERT model. We overcome this challenge by extracting dense- cap …

Content truncated to 500 words