Is an Image Worth More than a Thousand Words? On the Fine-Grain Semantic Differences between Visual and Linguistic Representations

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2807–2817, Osaka, Japan, December 11-17 2016. Is an Image Worth More than a Thousand Words? On the Fine-Grain Semantic Differences between Visual and Linguistic Representations Guillem Collell Computer Science Department KU Leuven 3001 Heverlee, Belgium gcollell@kuleuven.be Marie-Francine Moens Computer Science Department KU Leuven 3001 Heverlee, Belgium sien.moens@cs.kuleuven.be Abstract Human concept representations are often grounded with visual information, yet some aspects of meaning cannot be visually represented or are better described with language. Thus, vision and language provide complementary information that, properly combined, can potentially yield more complete concept representations. Recently, state-of-the-art distributional semantic mod- els and convolutional neural networks have achieved great success in representing linguistic and visual knowledge respectively. In this paper, we compare both, visual and linguistic representa- tions in their ability to capture different types of fine-grain semantic knowledge—or attributes— of concepts. Humans often describe objects using attributes, that is, properties such as shape, color or functionality, which often transcend the linguistic and visual modalities. In our setting, we evaluate how well attributes can be predicted by using the unimodal representations as inputs. We are interested in first, finding out whether attributes are generally better captured by either the vision or by the language modality; and second, if none of them is clearly superior (as we hy- pothesize), what type of attributes or semantic knowledge are better encoded from each modality. Ultimately, our study sheds light on the potential of combining visual and textual representations. 1 Introduction Vision and language capture complementary information that humans automatically integrate in order to build mental representations of concepts. Certain concepts or properties of objects cannot be explicitly visually represented while, at the same time, not all the properties are easily expressible with language. For example, there are clearly visual differences between cats and dogs although these are not easy to describe with language. Recent advances in deep learning had led to breakthroughs in learning unimodal representations (a.k.a. embeddings) in both, computer vision (CV) and natural language processing (NLP) (LeCun et al., 2015). However, the automatic integration of visual and linguistic modalities is still a challenging—and usually task-dependent—problem that has gained increasing popularity within the NLP and CV communities. Lately, several studies have achieved reasonable success in integrating visual and linguistic representations, showing improvement over the unimodal baselines in simple linguistic tasks such as concept similarity (Lazaridou et al., 2015; Kiela and Bottou, 2014; Silberer and Lapata, 2014)—which is only possible if vision and language encode complementary knowledge. In this paper we do not tackle the problem of how to integrate both modalities, but instead, we sys- tematically study what type of fine-grain semantic knowledge is encoded in each modality, shedding light on the potential benefit of combining vision and language. By fine-grain semantics we refer to the recognition of different types of attributes or properties (e.g., shape, function, sound, etc.) that concrete nouns might exhibit. A recent study by Rubinstein et al. (2015) evidenced that state-of-the-art …

Content truncated to 500 words