Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case

Proceedings of the 28th International Conference on Computational Linguistics, pages 730–744 Barcelona, Spain (Online), December 8-13, 2020 730 Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case Adam Dahlgren Lindstr¨om, Suna Bensch∗, Johanna Bj¨orklund∗, Frank Drewes∗ Department of Computing Science, Ume˚a University (Sweden) {dali, suna, johanna, drewes}@cs.umu.se Abstract Semantic embeddings have advanced the state of the art for countless natural language processing tasks, and various extensions to multimodal domains, such as visual-semantic embeddings, have been proposed. While the power of visual-semantic embeddings comes from the distillation and enrichment of information through machine learning, their inner workings are poorly understood and there is a shortage of analysis tools. To address this problem, we generalize the notion of probing tasks to the visual-semantic case. To this end, we (i) discuss the formalization of probing tasks for embeddings of image-caption pairs, (ii) define three concrete probing tasks within our general framework, (iii) train classifiers to probe for those properties, and (iv) compare various state-of-the-art embeddings under the lens of the proposed probing tasks. Our experiments reveal an up to 12% increase in accuracy on visual-semantic embeddings compared to the corresponding unimodal embeddings, which suggest that the text and image dimensions represented in the former do complement each other. 1 Introduction Semantic analysis aims to infer meaning from data. In a mathematical sense, the analysis relates ob- jects in a syntactic domain to objects in a semantic domain. In natural language processing, semantic embeddings such as word2vec and, more recently, BERT and GPT-2, have had a revolutionary impact on semantic analysis. The embeddings map words to real-valued vectors which reveal semantic aspects, for example, if words are related in meaning or belong to the same topic. Creating such an embedding means to enrich as well as filter out information. Unavoidably, some (usually surface and syntactic) information will be lost in the process of projecting words and their contexts onto a representation that focuses on meaning. Hence, there is a trade-off: the better we capture the semantics, the more surface and syntactic information becomes blurred. It depends on the downstream task what is the right balance between abstraction and detail. In multimodal semantic analysis, the syntactic domain is a Cartesian product of two or more domains, such as a video with audio tracks. This type of semantic analysis is increasingly applicable, showcas- ing convincing applications and results. Multimodal learning models such as DeViSE (Frome et al., 2013) demonstrate in particular that zero-shot learning can be signficantly improved by engaging mul- tiple modalities. There are many models based on machine learning techniques that jointly process the input modalities (Shen et al., 2019; Wu et al., 2019). The combination of modalities that has hitherto received the greatest interest is the pairing of images and text. Semantic embeddings of such data are commonly called visual-semantic embeddings. Throughout the rest of this paper, we focus on visual- semantic embeddings, and base the empirical part of our work on the dataset Common Objects in Context (MS-COCO), which consists of images …

Content truncated to 500 words