Multilingual Image Corpus – Towards a Multimodal and Multilingual Dataset

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1509–1518 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1509 Multilingual Image Corpus – Towards a Multimodal and Multilingual Dataset Svetla Koeva1, Ivelina Stoyanova2, Jordan Kralev3 1,2,3 Institute for Bulgarian Language, Bulgarian Academy of Sciences 3Technical University, Sofia 1svetla@dcl.bas.bg, 2iva@dcl.bas.bg, 3kralev@dcl.bas.bg Abstract One of the processing tasks for large multimodal data streams is the automatic image description (image classification, object segmentation and classification). Although the number and the diversity of image datasets is constantly expanding, there is still a huge demand for more datasets in terms of variety of domains and object classes covered. The goal of the project Multilingual Image Corpus (MIC21) is to provide a large image dataset with annotated objects and object descriptions in 25 languages. The Multilingual Image Corpus relies on an Ontology of Visual Objects (based on WordNet) and comprises a collection of thematically related images whose objects are annotated with segmentation masks and labels linked to the ontology classes. The dataset is designed both for image classification and object detection and for semantic segmentation. The main contributions of our work are: a) the provision of a large collection of high-quality images licensed for commercial and non-commercial use; b) the compilation of the Ontology of Visual Objects based on WordNet noun hierarchies; c) the automatic object segmentation within the images followed by precise manual editing and the annotation of object classes; and d) the mapping of objects and images to extended multilingual descriptions based on WordNet inner- and interlingual relations. The dataset can be used also for multilingual image caption generation, image-to-text alignment and automatic question answering for multimedia content. Keywords: multilingual image corpus, multilingual dataset, multimodal dataset 1. Introduction Nowadays the multimodal content combining text, im- ages and video is rapidly increasing. The processing of big multimodal data based on reliable computer vision models poses ever increasing challenges on the quality and the amount of the training data. Although the size of the image datasets can be artificially increased by synthetic data, the synthetic data has to be grounded by real-world images and their annotations. The multimodal data sources supply richer informa- tion than a single-mode input. Advanced technologi- cal solutions for analysing multimodal content can be provided using computer vision methods together with natural language processing. There has been signifi- cant progress in many multimodal tasks such as im- age caption generation (Xu et al., 2016), aligning sen- tences with images in various types of multimodal doc- uments (Hessel et al., 2019) and visual question an- swering (Antol et al., 2015; Sethuraman et al., 2021; Le et al., 2020; Gao et al., 2020). These tasks can be achieved by interpreting an image and the correspond- ing short text such as a caption, question or description of the objects in the image. The shift of traditional data fusion methods challenged by big multimodal data motivates the creation of a new image corpus, the Multilingual Image Corpus …

Content truncated to 500 words