Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings

Proceedings of NAACL-HLT 2016, pages 182–192, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings Spandana Gella, Mirella Lapata and Frank Keller Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB S.Gella@sms.ed.ac.uk, mlap@inf.ed.ac.uk, keller@inf.ed.ac.uk Abstract We introduce a new task, visual sense disam- biguation for verbs: given an image and a verb, assign the correct sense of the verb, i.e., the one that describes the action depicted in the image. Just as textual word sense disambigua- tion is useful for a wide range of NLP tasks, visual sense disambiguation can be useful for multimodal tasks such as image retrieval, im- age description, and text illustration. We intro- duce VerSe, a new dataset that augments exist- ing multimodal datasets (COCO and TUHOI) with sense labels. We propose an unsupervised algorithm based on Lesk which performs vi- sual sense disambiguation using textual, vi- sual, or multimodal embeddings. We find that textual embeddings perform well when gold- standard textual annotations (object labels and image descriptions) are available, while mul- timodal embeddings perform well on unanno- tated images. We also verify our findings by using the textual and multimodal embeddings as features in a supervised setting and analyse the performance of visual sense disambigua- tion task. VerSe is made publicly available and can be downloaded at: https://github. com/spandanagella/verse. 1 Introduction Word sense disambiguation (WSD) is a widely stud- ied task in natural language processing: given a word and its context, assign the correct sense of the word based on a pre-defined sense inventory (Kilgarrif, 1998). WSD is useful for a range of NLP tasks, including information retrieval, information extrac- tion, machine translation, content analysis, and lex- icography (see Navigli (2009) for an overview). Figure 1: Visual sense ambiguity: three of the senses of the verb play. Standard WSD disambiguates words based on their textual context; however, in a multimodal setting (e.g., newspaper articles with photographs), visual context is also available and can be used for disam- biguation. Based on this observation, we introduce a new task, visual sense disambiguation (VSD) for verbs: given an image and a verb, assign the correct sense of the verb, i.e., the one depicted in the image. While VSD approaches for nouns exist, VSD for verbs is a novel, more challenging task, and related in interesting ways to action recognition in computer vision. As an example consider the verb play, which can have the senses participate in sport, play on an instrument, and be engaged in playful activity, de- pending on its visual context, see Figure 1. We expect visual sense disambiguation to be use- ful for multimodal tasks such as image retrieval. As an example consider the output of Google Image Search for the query sit: it recognizes that the verb has multiple senses and tries to cluster relevant im- ages. However, the result does not capture the pol- ysemy of the verb well, and would clearly benefit …

Content truncated to 500 words