Fatality Killed the Cat or: BabelPic, a Multimodal Dataset for Non-Concrete Concepts

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4680–4686 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4680 Fatality Killed the Cat or: BabelPic, a Multimodal Dataset for Non-Concrete Concepts Agostina Calabrese, Michele Bevilacqua and Roberto Navigli Sapienza NLP Group Department of Computer Science Sapienza University of Rome {calabrese.a,bevilacqua,navigli}@di.uniroma1.it Abstract Thanks to the wealth of high-quality annotated images available in popular repositories such as ImageNet, multimodal language-vision re- search is in full bloom. However, events, feel- ings and many other kinds of concepts which can be visually grounded are not well repre- sented in current datasets. Nevertheless, we would expect a wide-coverage language un- derstanding system to be able to classify im- ages depicting RECESS and REMORSE, not just CATS, DOGS and BRIDGES. We fill this gap by presenting BabelPic, a hand-labeled dataset built by cleaning the image-synset association found within the BabelNet Lexical Knowledge Base (LKB). BabelPic explicitly targets non- concrete concepts, thus providing refreshing new data for the community. We also show that pre-trained language-vision systems can be used to further expand the resource by ex- ploiting natural language knowledge available in the LKB. BabelPic is available for down- load at http://babelpic.org. 1 Introduction There is growing research interest in developing effective systems capable of achieving some under- standing of the content of an image. As in most fields of applied AI, this requires annotated data to train a supervised system on. While ImageNet1 (Deng et al., 2009), one of the most influential projects in computer vision, was undeniably an important milestone towards image understanding, there is still a lot of ground to be covered. Ima- geNet’s initial aim was to collect pictures for most WordNet synsets (Miller, 1995). Yet, at the time of writing, only some 21,841 nominal synsets are covered according to ImageNet’s official website. One issue with ImageNet and most other im- age repositories like COCO (Lin et al., 2014) and 1http://www.image-net.org Flickr30kEntities (Plummer et al., 2015) is their fo- cus on concepts denoting concrete, tangible things, such as CAT, TRAFFIC LIGHT and so on. Concepts whose denotation is not clearly identifiable with a set of objects having distinct boundaries, such as events (e.g., FATALITY, COMPETITION), emotions (e.g., SADNESS) and psychological features (e.g., SHARPNESS), have enjoyed less attention. For lack of a better term, we will henceforth refer to them as non-concrete (NC) concepts. On one hand, the inclusion of NC concepts would be an important step towards wide-coverage image semantic understanding. On the other hand, it also goes in the same direction as recent mul- timodal language-vision approaches, e.g., mono- and cross-lingual Visual Sense Disambiguation (Barnard and Johnson, 2005; Loeff et al., 2006; Saenko and Darrell, 2008; Gella et al., 2016, 2019). Taking into account NC concepts could also be of crucial importance for fascinating language- focused applications, such as Multimodal Machine Translation. Last but not least, NC concepts would represent a significative benchmark for real-world multimodal applications. In fact, traditional com- puter vision approaches rely on …

Content truncated to 500 words