Sketchy Scene Captioning: Learning Multi-Level Semantic Information from Sparse Visual Scene Cues

Sketchy Scene Captioning: Learning Multi-Level Semantic Information from Sparse Visual Scene Cues Lian Zhou, Yangdong Chen, Yuejie Zhang† School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai 200438, China {16110240019, 19110240010, yjzhang}@fudan.edu.cn Abstract To enrich the research about sketch modality, a new task termed Sketchy Scene Captioning is proposed in this paper. This task aims to generate sentence-level and paragraph-level descrip- tions for a sketchy scene. The sentence-level description provides the salient semantics of a sketchy scene while the paragraph-level description gives more details about the sketchy scene. Sketchy Scene Captioning can be viewed as an extension of sketch classification which can only provide one class label for a sketch. To generate multi-level descriptions for a sketchy scene is challenging because of the visual sparsity and ambiguity of the sketch modality. To achieve our goal, we first contribute a sketchy scene captioning dataset to lay the foundation of this new task. The popular sequence learning scheme, e.g., Long Short-Term Memory neural network with vi- sual attention mechanism, is then adopted to recognize the objects in a sketchy scene and infer the relations among the objects. In the experiments, promising results have been achieved on the proposed dataset. We believe that this work will motivate further researches on the understanding of sketch modality and the numerous sketch-based applications in our daily life. The collected dataset is released at https://github.com/SketchysceneCaption/Dataset. 1 Introduction In recent years, sketch has emerged as one important data modality (Eitz et al., 2012; Yu et al., 2015). Compared to a natural image, a sketch only contains sparse and ambiguous visual information. Current works about sketch mainly focus on predicting one class label for a sketch, and such a label provides very limited semantic information (Eitz et al., 2012). Differently, the tasks about natural image are abundant, such as classification (Deng et al., 2009), captioning (Vinyals et al., 2015; Xu et al., 2015; Anderson et al., 2018), and visual question answering (Anderson et al., 2018). What hinders the research about sketch is the lack of sketch datasets. Specifically, natural images are easy to be obtained and a lot of efforts have been put into annotating the images. By contrast, sketch is created by human and the generation of a sketch is time-consuming, which limits the volume of a sketch dataset and the visual details of the sketches in the dataset. Hence, most of current sketch datasets only contain sketches with a single object and the corresponding class label. Drawing inspiration from the task of natural image captioning, it is attractive to expand a sketch to a sketchy scene which contains several objects, and extend the class label to a sentence-level or even a paragraph-level description. In a word, a sketchy scene dataset with multi-level descriptions is in urgent need to promote the research about sketch. One promising application with above extension is child education (Neshati et al., 2017). Specifically, with the wide popularity of tablet PC, it becomes common for a child to doodle …

Content truncated to 500 words