Edit me: A Corpus and a Framework for Understanding Natural Language Image Editing

Edit me: A Corpus and a Framework for Understanding Natural Language Image Editing Ramesh Manuvinakurike∗2, Jacqueline Brixey∗2, Trung Bui1, Walter Chang1, Doo Soon Kim1, Ron Artstein2, Kallirroi Georgila2 1Adobe Research, San Jose, USA 2USC Institute for Creative Technologies, Los Angeles, USA {manuvina,brixey}@usc.edu, {bui,wachang,dkim}@adobe.com, {artstein,kgeorgila}@ict.usc.edu Abstract This paper introduces the task of interacting with an image editing program through natural language. We present a corpus of image edit requests which were elicited for real world images, and an annotation framework for understanding such natural language instructions and mapping them to actionable computer commands. Finally, we evaluate crowd-sourced annotation as a means of efficiently creating a sizable corpus at a reasonable cost. Keywords: dialogue, image editing, vision and language 1. Introduction Photo editing is as old as photography itself. Over the years, darkroom techniques with film and light have given way to digital processing, and software suites such as Adobe Photoshop have made image editing accessible to millions of professionals and hobbyists. But even with the best tools, photo editing requires substantial knowledge, and novices often need to enlist the help of experts. Web sites such as https://www.reddit.com/r/PhotoshopRequest/ and http://zhopped.com/ contain thousands of image edit requests like the following: • There is a spot on my wedding dress. Can someone please remove it. Please! • Can you please fix the glare on my dog’s eyes? I lost him today and he means the world to me. • Can you please remove the people in the background? This is the only surviving photo of my mom and I would like to preserve it. • I love this photo from our trip to Rome. Can someone please remove my ex from this photo? I am the one on the right. As the examples above show, a natural way for novices to express their editing needs is through ordinary human lan- guage. At Adobe Research we aim to develop a software tool that will interpret such natural language image edit re- quests, and carry them out to the user’s satisfaction. This work presents a step in the direction of understand- ing human image edit requests: a corpus of such requests, and an annotation scheme for mapping these requests into actionable computer commands. A corpus could be com- piled from naturally occurring examples such as the ones cited above, but this would raise concerns about privacy and ownership of the photographs, and the images themselves are unprocessed. Instead, we use a set of publicly available, richly annotated images called the Visual Genome corpus (Krishna et al., 2017). We elicited image edit requests that * Primary authors; work done while at Adobe Research would pertain to these images, devised a scheme for map- ping these requests into actionable commands, annotated requests according to the scheme, and evaluated the relia- bility of these annotations. The contributions of this work are threefold. First, a data- set of natural language image edit requests for images with detailed image captions. These captions are particularly useful for the task of language understanding, …

Content truncated to 500 words