FEIDEGGER: A Multi-modal Corpus of Fashion Images and Descriptions in German

FEIDEGGER: A Multi-modal Corpus of Fashion Images and Descriptions in German Leonidas Lefakis, Alan Akbik, Roland Vollgraf Zalando Research M¨uhlenstraße 25, 10243 Berlin {firstname.lastname}@zalando.de Abstract The availability of multi-modal datasets that pair images and textual descriptions of their content has been a crucial driver in progress of various text-image tasks such as automatic captioning and text-to-image retrieval. In this paper, we present FEIDEGGER, a new multi-modal corpus that focuses specifically on the domain of fashion items and their visual descriptions in German. We argue that such narrow-domain multi-modality presents a unique set of challenges such as fine-grained image distinctions and domain-specific language, and release this dataset to the research community to enable study of these challenges. This paper illustrates our crowdsourcing strategy to acquire the textual descriptions, gives an overview over the FEIDEGGER dataset, and discusses possible use cases. Keywords: Crowdsourcing, Multi-modalily, Fashion, German 1. Introduction Recent years have seen a renewed interest in text-image multi-modality and have seen the emergence of tasks such as automatically generating textual captions for a given im- age (Karpathy and Fei-Fei, 2015; Lu et al., 2016), using plain text descriptions to query images (Socher et al., 2014), and matching lexical tokens or constituents to regions in an image (Ma et al., 2015). This interest is driven by advances in multi-modal deep learning for computer vision (Vinyals et al., 2015; Karpathy et al., 2014) on the one hand, as well as the availability of paired text-image datasets on the other. Multi-modal text-image datasets. Commonly cited multi- modal datasets either consist of user-captioned images from the web, such as FLICKR (Hodosh et al., 2013; Plummer et al., 2015) and online news (Hollink et al., 2016), or im- ages for which crowd workers have produced visual de- scriptions (Rashtchian et al., 2010), such as the popular COCO dataset (Lin et al., 2014). In both cases, the tex- tual data directly describes image content, thus enabling the above-mentioned lines of research. However, these datasets are often restricted to English lan- guage text and typically of relatively broad domain; The FLICKR caption datasets for instance contain images in- cluding landscapes, animals, and everyday scenes while the COCO dataset is similarly broad but contains more items per image. This makes such datasets difficult to apply for study of multi-modality in more narrow domains. In the domain of fashion items for instance, images are broadly similar and are often distinguished only by fine-grained dif- ferences such as the material, the neckline, brand logos, the cut and the style of the hem. Similarly, the language used in fashion is domain-specific, tailored specifically to highlight such fine-grained differences. We argue that such narrow domains present unique research challenges that re- quire specialized multi-modal datasets. A multi-modal text-image corpus for fashion. With this paper, we introduce a novel dataset for research in narrow- domain multi-modality, called FEIDEGGER1. Contrary to 1A rough acronym of ”fashion image data and descriptions in previous datasets, we restrict the domain to images of one type of fashion item, namely …

Content truncated to 500 words