Face2Text: Collecting an Annotated Image Description Corpus for the Generation of Rich Face Descriptions

Face2Text: Collecting an Annotated Image Description Corpus for the Generation of Rich Face Descriptions Albert Gatt∗, Marc Tanti∗, Adrian Muscat∗, Patrizia Paggio∗†, Reuben A Farrugia∗, Claudia Borg∗, Kenneth P Camilleri∗, Michael Rosner∗, Lonneke van der Plas∗ ∗University of Malta {albert.gatt, marc.tanti.06, adrian.muscat, patrizia.paggio, reuben.farrugia}@um.edu.mt {claudia.borg, kenneth.camilleri, mike.rosner, lonneke.vanderplas}@um.edu.mt †University of Copenhagen paggio@hum.ku.dk Abstract The past few years have witnessed renewed interest in NLP tasks at the interface between vision and language. One intensively-studied problem is that of automatically generating text from images. In this paper, we extend this problem to the more specific domain of face description. Unlike scene descriptions, face descriptions are more fine-grained and rely on attributes extracted from the image, rather than objects and relations. Given that no data exists for this task, we present an ongoing crowdsourcing study to collect a corpus of descriptions of face images taken ‘in the wild’. To gain a better understanding of the variation we find in face description and the possible issues that this may raise, we also conducted an annotation study on a subset of the corpus. Primarily, we found descriptions to refer to a mixture of attributes, not only physical, but also emotional and inferential, which is bound to create further challenges for current image-to-text methods. Keywords: face images, vision and language, image-to-text, Natural Language Generation, NLG, crowdsourcing 1. Introduction This paper describes an annotation project that is being con- ducted by a cross-disciplinary group of researchers at the University of Malta, the RIVAL (Research In Vision And Language) group, to create a corpus of human face images annotated with rich textual descriptions. The initial goal of the project was to investigate in general how users describe images of human faces, and ultimately to create a resource that could be used to train a system to generate descriptions potentially as varied and rich as possible, thus moving the state-of-the-art of automatic description generation for im- ages of faces from a feature-based process to one that takes advantage of complex textual descriptions. Here we report on a preliminary version of the corpus, focussing on how it was collected and evaluated.1 2. Background Automatic image description research can rely on a wide range of image-description datasets. Such datasets con- sist of images depicting various objects and actions, and associated descriptions, typically collected through crowd- sourcing. The descriptions verbalise the objects and events or relations shown in the images with different degrees of granularity. For example, the most widely-used image captioning datsets, such as Flickr8k (Hodosh et al., 2013), Flickr30K (Young et al., 2014), VLT2K (Elliott and Keller, 2013), and MS COCO (Lin et al., 2014), contain images of familiar scenes, and the descriptions are restricted to the ‘concrete conceptual’ level (Hodosh et al., 2013), men- tioning what is visible, while minimising inferences that can be drawn from the visual information. Other datasets are somewhat more specialised. For example, the Caltech- UCSD Birds and Oxford Flowers-102 contain fine-grained 1The corpus will shortly be released to the public. The current version is …

Content truncated to 500 words