Polish Corpus of Annotated Descriptions of Images

Polish Corpus of Annotated Descriptions of Images Alina Wr´oblewska Institute of Computer Science, Polish Academy of Sciences Warsaw, Poland alina@ipipan.waw.pl Abstract The paper presents a new dataset of image descriptions in Polish. The descriptions are morphosyntactically analysed and the pairs of these descriptions are annotated in terms of semantic relatedness and entailment. All annotations are provided by human annotators with strong linguistic background. The dataset can be used for evaluation of various systems integrating language and vision. It is applicable for evaluation of systems designed to image generation based on provided descriptions (text-to-image generation) or to caption generation based on images (image-to-text generation). Furthermore, as selected images are split into thematic groups, the dataset is also useful for validating image classification approaches. Keywords: language and vision, evaluation dataset, annotated image descriptions, Polish 1. Introduction Language and vision are two essential modalities that en- able interpersonal communication. As broadly understood communication is an important area of artificial intelli- gence, AI researchers attach importance to image process- ing, natural language processing, and integration of these two fields. Vision-to-language approaches consist mostly in mapping images to sentences, e.g. Hodosh et al. (2013), or generating image descriptions, e.g. Xu et al. (2015), Karpathy and Fei-Fei (2017). For example, Karpathy and Fei-Fei (2017) propose a model that generates natural language descriptions of whole images or their regions. Based on datasets of images and their sentence descrip- tions, the model learns about the inter-modal correspon- dences between language and vision. Regarding language- to-vision approaches, research and scientific experiments are conducted in the areas of text-based image retrieval, e.g. Rasiwasia et al. (2010), or text-based image generation, e.g. Denton et al. (2015). For example, the paper by Denton et al. (2015) describes a “generative parametric model capable of producing high quality samples of natural images”. In or- der to evaluate many of these approaches, the high-quality caption-image datasets are necessary. This paper presents a new Polish dataset of annotated im- age descriptions – AIDe1 (Annotated Image Descriptions). The dataset consists of 2K natural language descriptions of 1K images. The dataset is probably too small for training a sophisticated language–vision system. For training pur- poses, the dataset should be expanded to the greatest pos- sible extent. However, in the age of intensive research on multilingual NLP, e.g. Faruqui and Dyer (2014), it seems to make sense to build even small but high-quality evalua- tion resources. The presented dataset can be used for evaluation of vari- ous systems integrating language and vision. It is applicable for evaluation of systems designed to generation of images based on provided descriptions (text-to-image generation) or to generation of captions based on images (image-to-text generation). Furthermore, as elected images are split into 1http://zil.ipipan.waw.pl/Scwad/AIDe thematic groups, the dataset is also useful for validating im- age classification approaches. The procedure of selecting, describing, and splitting images into thematic groups is described in Section 2. The image descriptions are morphosyntactically annotated (see Sec- tion 3.) and the pairs of these descriptions are annotated in terms of semantic …

Content truncated to 500 words