A Visually-Grounded Parallel Corpus with Phrase-to-Region Linking

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4204–4210 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4204 A Visually-Grounded Parallel Corpus with Phrase-to-Region Linking Hideki Nakayama1, Akihiro Tamura2, Takashi Ninomiya2 1Graduate School of Information Science and Technology, The University of Tokyo 2Graduate School of Science and Engineering, Ehime University nakayama@ci.i.u-tokyo.ac.jp, {tamura, ninomiya}@cs.ehime-u.ac.jp Abstract Visually-grounded natural language processing has become an important research direction in the past few years. However, majorities of the available cross-modal resources (e.g., image-caption datasets) are built in English and cannot be directly utilized in multilingual or non-English scenarios. In this study, we present a novel multilingual multimodal corpus by extending the Flickr30k Entities image- caption dataset with Japanese translations, which we name Flickr30k Entities JP (F30kEnt-JP). To the best of our knowledge, this is the first multilingual image-caption dataset where the captions in the two languages are parallel and have the shared annotations of many- to-many phrase-to-region linking. We believe that phrase-to-region as well as phrase-to-phrase supervision can play a vital role in fine- grained grounding of language and vision, and will promote many tasks such as multilingual image captioning and multimodal machine translation. To verify our dataset, we performed phrase localization experiments in both languages and investigated the effectiveness of our Japanese annotations as well as multilingual learning realized by our dataset. Keywords: Multimodal Parallel Corpus, Visual Grounding, Phrase-to-Region Matching, Multimodal Machine Translation 1. Introduction Grounding natural language to other modalities has re- cently been gaining much attention as a promising ap- proach to go beyond traditional NLP by incorporating information or knowledge not readily available in tex- tual domain. Particularly, visual grounding has be- come a popular topic because of the richness and im- portance of visual information in our perception of the world, and many so-called language and vision tasks have been proposed. Representative examples in- clude image and video captioning (Vinyals et al., 2015), visual question answering (Agrawal et al., 2015), vi- sual story telling (Huang et al., 2016), visual dialog (Das et al., 2017), and multimodal machine translation (Elliott et al., 2015; Hitschler et al., 2016). To promote the research based on visual grounding, multi- modal corpus, i.e. a set of pairs of an image and a natural language description depicting the same underlying con- cept, has been an indispensable resource. However, ex- isting multimodal corpora have limitations in the follow- ing aspects. First, most of the existing large-scale cor- pora are monolingual and built in English (Lin et al., 2014; Krishna et al., 2017; Sharma et al., 2018). While they are useful for language and vision problems in general, they cannot be directly used for the study of cross-lingual tasks such as machine translation, and specific problems in each non-English language. Second, although there exist some multilingual multimodal corpora that cover more than one language, the link between images and texts is not very strong in the sense that only global-level (i.e., sentence-to- image) correspondence is available. To realize …

Content truncated to 500 words