MultiSubs: A Large-scale Multimodal and Multilingual Dataset

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6776–6785 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6776 MultiSubs: A Large-scale Multimodal and Multilingual Dataset Josiah Wang1, Josiel Figueiredo2, Lucia Specia1,3 1Imperial College London, UK 2Federal University of Mato Grosso, Brazil 3University of Sheffield, UK josiah.wang@imperial.ac.uk, josiel@ic.ufmt.br, l.specia@imperial.ac.uk Abstract This paper introduces a large-scale multimodal and multilingual dataset that aims to facilitate research on grounding words to images in their contextual usage in language. The dataset consists of images selected to unambiguously illustrate concepts expressed in sentences from movie subtitles. The dataset is a valuable resource as (i) the images are aligned to text fragments rather than whole sentences; (ii) multiple images are possible for a text fragment and a sentence; (iii) the sentences are free-form and real-world like; (iv) the parallel texts are multilingual. We also set up a fill-in-the-blank game for humans to evaluate the quality of the automatic image selection process of our dataset. Finally, we propose a fill-in-the-blank task to demonstrate the utility of the dataset, and present some baseline prediction models. The dataset will benefit research on visual grounding of words especially in the context of free-form sentences, and can be obtained from https://doi.org/10.5281/zenodo.5034604 under a Creative Commons licence. Keywords: Multimodality, Visual Grounding, Multilinguality, Multimodal Dataset 1. Introduction “Our experience of the world is multimodal – we see objects, hear sounds, feel texture, smell odours, and taste flavours” (Baltruˇsaitis et al., 2019). In order to understand the world around us, we need to be able to interpret such multimodal signals together. Learning and understanding languages is not an exception: hu- mans make use of multiple modalities when doing so. In particular, words are generally learned with visual (among others) input as additional modality. Research on computational models of language grounding using visual information has led to many interesting applica- tions, such as Image Captioning (Vinyals et al., 2015), Visual Question Answering (Antol et al., 2015) and Vi- sual Dialog (Das et al., 2017). Various multimodal datasets comprising images and text have been constructed for different applications. Many of these are made up of images annotated with text labels, and thus do not provide a context in which to apply the text and/or images. More recent datasets for image captioning (Chen et al., 2015; Young et al., 2014) go beyond textual labels and annotate images with sentence-level text. While these sentences pro- vide a stronger context for the image, they suffer from one primary shortcoming: Each sentence ‘explains’ an image given as a whole, while most often focusing on only some of the elements depicted in the image. This makes it difficult to learn correspondences between el- ements in the text and their visual representation. In- deed, the connection between images and text is multi- faceted, i.e. the former is not strictly a representation of the latter, thus making it hard to describe a whole im- age in a single sentence or to illustrate a whole …

Content truncated to 500 words