VICTR: Visual Information Captured Text Representation for Text-to-Vision Multimodal Tasks

Proceedings of the 28th International Conference on Computational Linguistics, pages 3107–3117 Barcelona, Spain (Online), December 8-13, 2020 3107 VICTR: Visual Information Captured Text Representation for Text-to-Image Multimodal Tasks Soyeon Caren Han*,**, Siqu Long*, Siwen Luo, Kunze Wang, Josiah Poon School of Computer Science, The University of Sydney 1 Cleveland Street, NSW 2006, Australia {caren.han, slon6753, siwen.luo, kwan4418, josiah.poon}@sydney.edu.au Abstract Text-to-image multimodal tasks, generating/retrieving an image from a given text description, are extremely challenging tasks since raw text descriptions cover quite limited information in order to fully describe visually realistic images. We propose a new visual contextual text representation for text-to-image multimodal tasks, VICTR, which captures rich visual semantic information of objects from the text input. First, we use the text description as initial input and conduct dependency parsing to extract the syntactic structure and analyse the semantic aspect, including object quantities, to extract the scene graph. Then, we train the extracted objects, attributes, and relations in the scene graph and the corresponding geometric relation information using Graph Convolutional Networks, and it generates text representation which integrates textual and visual semantic information. The text representation is aggregated with word-level and sentence- level embedding to generate both visual contextual word and sentence representation. For the evaluation, we attached VICTR to the state-of-the-art models in text-to-image generation.VICTR is easily added to existing models and improves across both quantitative and qualitative aspects1. 1 Introduction Over the past decade, deep learning has achieved remarkable success in multimodal research problems, such as visual question answering, visual dialog, and image captioning. However, it is quite challenging to solve the text-to-vision multimodal tasks that deal with a text input and produce a visual output, such as text to image generation, text to video generation, or text to image retrieval. Generally, natural language (incl. text) is a more natural medium for a human to describe the image that they want to generate or retrieve. However, raw text includes only limited information to fully describe and represent an image. For example, text to image generation tasks aim to generate photo-realistic images according to the given text descriptions. The current state-of-the-art (SOTA) text-to-image generation models (Xu et al., 2018; Zhang et al., 2017; Zhu et al., 2019) mainly focus on generating the high resolution images by ap- plying generative adversarial networks (GAN) (Goodfellow et al., 2014) and rather neglect understanding input text descriptions. The most common text encoding approach in those SOTA models applies Re- current Neural Networks (RNN) to extract global sentence-level and word-level embedding, and the last hidden state of RNN cells is used as a direct input to the GAN image generation model. However, RNN- based text encoding approaches are not capable of fully representing the rich visual semantics of input text descriptions in order to describe or generate photo-realistic visual output (e.g. image). For example, from the given sample text caption, A man on a skateboard with a brown dog outside, the last hidden state of RNN cells holds semantics and order of words and stores words’ …

Content truncated to 500 words