Learning Visually-Grounded Semantics from Contrastive Adversarial Samples

Proceedings of the 27th International Conference on Computational Linguistics, pages 3715–3727 Santa Fe, New Mexico, USA, August 20-26, 2018. 3715 Learning Visually-Grounded Semantics from Contrastive Adversarial Samples Haoyue Shi 1∗ Jiayuan Mao 2∗ Tete Xiao 1∗ Yuning Jiang 3 Jian Sun 3 1: School of Electronics Engineering and Computer Science, Peking University, China 2: ITCS, Institute for Interdisciplinary Information Sciences, Tsinghua University, China 3: Megvii, Inc. {hyshi,jasonhsiao97}@pku.edu.cn mjy14@mails.tsinghua.edu.cn, {jyn, sunjian}@megvii.com Abstract We study the problem of grounding distributional representations of texts on the visual domain, namely visual-semantic embeddings (VSE for short). Begin with an insightful adversarial attack on VSE embeddings, we show the limitation of current frameworks and image-text datasets (e.g., MS-COCO) both quantitatively and qualitatively. The large gap between the number of possible constitutions of real-world semantics and the size of parallel data, to a large extent, restricts the model to establish a strong link between textual semantics and visual concepts. We alleviate this problem by augmenting the MS-COCO image captioning datasets with textual contrastive adver- sarial samples. These samples are synthesized using language priors of human and the WordNet knowledge base, and enforce the model to ground learned embeddings to concrete concepts within the image. This simple but powerful technique brings a noticeable improvement over the baselines on a diverse set of downstream tasks, in addition to defending known-type adversarial attacks. Codes are available at https://github.com/ExplorerFreda/VSE-C. 1 Introduction The visual grounding of language plays an indispensable role in our daily lives. We use language to name, refer, and describe objects, their properties and generally, visual concepts. Distributional semantics (e.g., global word embedding (Pennington et al., 2014)) based on large-scale corpora have shown great success in modeling the functionality and correlation of words in the natural language domain. This further contributes to the success in numerous natural language processing (NLP) tasks such as language modeling (Cheng et al., 2016; Inan et al., 2017), sentiment analysis (Cheng et al., 2016; Kumar et al., 2016), and reading comprehension (Cheng et al., 2016; Chen et al., 2016; Shen et al., 2017). However, effective and efficient grounding of distributional embeddings remains challenging. Being ignorant of the corresponding visual concepts, pure textual embeddings demonstrate inferior performances when incorporating with visual inputs. Typical tasks include image/video captioning, multi-modal retrieval/understanding, and visual reasoning, some of which are further extensively studied in the paper. Visual concept and its link with textual semantics, as a cognitive alignment, provide rich supervision to learning systems. Visual-Semantic Embedding (VSE) aims at building the bridge between natural language and the underlying visual world. Introduced by Kiros et al. (2014), the embedding spaces of both images and descriptive texts (captions) are jointly optimized and aligned. Nevertheless, even for large-scale datasets such as MS-COCO (Lin et al., 2014), the number of image-caption pairs are far less than the number of possible constitutions of real-world semantics, making the dataset inevitably sparse and biased. To reveal this, we begin with constructing textual adversarial samples to attack the state-of-the-art system VSE++ (Faghri et al., 2017). Specifically, we …

Content truncated to 500 words