Large Scale Multi-Lingual Multi-Modal Summarization Dataset

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3620–3632 May 2-6, 2023 ©2023 Association for Computational Linguistics Large Scale Multi-Lingual Multi-Modal Summarization Dataset Yash Verma* IISER Kolkata yashv7523@gmail.com Anubhav Jangra* Indian Institute of Technology Patna anubhav0603@gmail.com Raghvendra Kumar Indian Institute of Technology Patna raghvendra.kumar1004@gmail.com Sriparna Saha Indian Institute of Technology Patna sriparna@iitp.ac.in Abstract Significant developments in techniques such as encoder-decoder models have enabled us to rep- resent information comprising multiple modal- ities. This information can further enhance many downstream tasks in the field of infor- mation retrieval and natural language process- ing; however, improvements in multi-modal techniques and their performance evaluation require large-scale multi-modal data which of- fers sufficient diversity. Multi-lingual mod- eling for a variety of tasks like multi-modal summarization, text generation, and transla- tion leverages information derived from high- quality multi-lingual annotated data. In this work, we present the current largest multi- lingual multi-modal summarization dataset (M3LS), and it consists of over a million in- stances of document-image pairs along with a professionally annotated multi-modal sum- mary for each pair. It is derived from news articles published by British Broadcasting Corporation(BBC) over a decade and spans 20 languages, targeting diversity across five language roots, it is also the largest summa- rization dataset for 13 languages and consists of cross-lingual summarization data for 2 lan- guages. We formally define the multi-lingual multi-modal summarization task utilizing our dataset and report baseline scores from various state-of-the-art summarization techniques in a multi-lingual setting. We also compare it with many similar datasets to analyze the uniqueness and difficulty of M3LS. 1 1 Introduction The world we live in today is very diverse, with over 7,000+ languages spoken across the globe2. These languages have varying traits and are spoken by communities of various sizes depending upon *These authors contributed equally to this work. 1The dataset and code used in this work are made available at https://github.com/anubhav-jangra/M3LS. 2https://www.ethnologue.com/guides/ how-many-languages Table 1: Comparison of proposed dataset with existing large-scale multi-lingual and multi-modal datasets. Multi-lingual Summarization Datasets Dataset Name Dataset Size #Languages Domain XL-Sum (Hasan et al., 2021) 1M 44 News MLSUM (Scialom et al., 2020) 1.5M 5 News WikiLingua (Ladhak et al., 2020) 770K 18 Tutorials MLGSum (Wang et al., 2021) 1.1M 12 News M3LS (Ours) 1.1M 20 News Multi-modal Summarization Datasets Dataset Name Dataset Size Modalities Domain MSMO (Zhu et al., 2018) 314K Text + Image News E-DailyMail (Chen and Zhuge, 2018) 219K Text + Image News How2 (Sanabria et al., 2018) 190K Text + Video + Audio Multiple Domains MMSS (Li et al., 2018) 66K Text + Image News VMSMO (Li et al., 2020) 185K Text + Video + Audio Social Network M3LS (Ours) 1.1M Text + Image News the popularity of the language. For example, Man- darin consists of over 50,000 hanzi (characters) and is spoken by over 1.117 billion people3, while there exist languages like Rotokas, which is an indige- nous language spoken by about 4,320 people on the island of Bougainville, Papua New …

Content truncated to 500 words