SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4413–4422, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4413 SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning Tsu-Jui Fu†, Xin Eric Wang‡, Scott T. Grafton†, Miguel P. Eckstein†, William Yang Wang† †UC Santa Barbara ‡UC Santa Cruz tsu-juifu@ucsb.edu, {scott.grafton, miguel.eckstein}@psych.ucsb.edu william@cs.ucsb.edu, xwang366@ucsc.edu Abstract Iterative Language-Based Image Editing (IL- BIE) tasks follow iterative instructions to edit images step by step. Data scarcity is a sig- nificant issue for ILBIE as it is challenging to collect large-scale examples of images be- fore and after instruction-based changes. How- ever, humans still accomplish these editing tasks even when presented with an unfamil- iar image-instruction pair. Such ability results from counterfactual thinking and the ability to think about alternatives to events that have hap- pened already. In this paper, we introduce a Self-Supervised Counterfactual Reasoning (SSCR) framework that incorporates counter- factual thinking to overcome data scarcity. SSCR allows the model to consider out-of- distribution instructions paired with previous images. With the help of cross-task consis- tency (CTC), we train these counterfactual in- structions in a self-supervised scenario. Ex- tensive results show that SSCR improves the correctness of ILBIE in terms of both object identity and position, establishing a new state of the art (SOTA) on two IBLIE datasets (i- CLEVR and CoDraw). Even with only 50% of the training data, SSCR achieves a compa- rable result to using complete data. 1 Introduction Digital design tools like Illustrator or Photoshop are widely used nowadays. Though having consid- erable user demand, they require prior knowledge and multiple steps to use successfully. These appli- cations would significantly improve the accessibil- ity if they can automatically perform corresponding editing actions based on the language instructions given by users for each step. Iterative language-based image editing (ILBIE) task follows iterative instructions to edit images step by step, as illustrated in Fig. 1. To accomplish ILBIE, models are required not only to modify im- ages but also to understand the visual differences Figure 1: An example of the iterative language-based image editing (ILBIE) task. During each turn, the model edits the image from the previous turn based on the current instruction. Eventually, a desired image is accomplished after iterative editing. Note that the gen- eration is at the pixel level. between the previous and resulting image, based on the given instructions. One of the primary limi- tations of ILBIE is data scarcity. Since collecting large-scale previous-resulting images with instruc- tions is difficult, it makes learning the association between vision and language challenging. A GAN-based (Goodfellow et al., 2015) model, GeNeVA (El-Nouby et al., 2019), is proposed to perform ILBIE, where a conditional genera- tor serves as the image editor, and a discrimina- tor provides the training loss by discriminating a resulting image. Though it yields promising re- sults, GeNeVA neglects the data scarcity issue of ILBIE. As a binary classifier, the discriminator eas- ily suffers from data scarcity given the shortage of ground-truth instruction-and-resulting-image pairs and …

Content truncated to 500 words