Multimodality for NLP-Centered Applications: Resources, Advances and Frontiers

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6837–6847 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6837 Multimodality for NLP-Centered Applications: Resources, Advances and Frontiers Muskan Garg*, **, Seema Wazarkar**, Muskaan Singh$,#, Ondˇrej Bojar$ * University of Florida, USA, # Speech and Audio Processing Group, IDIAP Research Institute, Switzerland ** Thapar Institute of Engineering & Technology, India, $ Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Czechia muskangarg@ufl.edu, msingh@idiap.ch, seema.wazarkar@thapar.edu, bojar@ufal.cuni.cz Abstract With advancements in the methods of Natural Language Processing (NLP) by explicitly considering other modalities than just text, the resurgence of multimodal datasets has attracted significant attention. However, there remains lack of a comprehensive survey on available datasets. To this end, we take the first step and present a thorough review of publicly available datasets with different modalities for NLP tasks which they may cater. Our survey shall enable the research community to re-use, re-furnish and re-annotate the existing datasets with new modalities for multiple NLP tasks. Furthermore, we discuss new frontiers and and challenges, and hope this survey will provide the community with a general picture of available multimodal datasets for various NLP applications, facilitate quick access to them and motivate future research. In this context, we release the collection of links to all multimodal datasets we discover as an easily accessible and updatable repository: https://github.com/ drmuskangarg/Multimodal-datasets 1. Introduction Multimodality refers to the capability of a system or method to process input of different types (or “modal- ities”), primarily text, image, sound or video. Em- braced with multiple streams of participants’ physical responses (eyetracking, EEG, etc.) or environmental conditions (temperature, pressure etc.), multimodality plays pivotal role in enhancing intelligence of a system. These multiple modalities (Parcalabescu et al., 2021) develop as a strong research enhancement in recent years to support downstream NLP tasks. We focus on the most common modalities in current NLP tasks and speak of 10 different permutations of four modalities as summarized in Figure 1. It is interesting that V (video) modality automatically leads to multiple combination of all other modalities for analysis. Research in this novel direction primarily aims to pro- cess textual content using visual information (e.g., im- ages and possibly video) to support various tasks (e.g., machine translation). Its motivation derives mainly from two linguistic challenges: lexical ambiguity and out of vocabulary words which may be resolved by using multiple modalities or stand for the missing in- formation in a way. In practice, the non-textual con- text provided implicitly by the additional modalities is extremely influential (“an image is worth a thousand words”, and a “sound illustration” can easily explain why, e.g., a person is having difficulties in express- ing themselves). Recent studies show that visual infor- mation helps in reaching modest but encouraging im- provements in quality (Elliott et al., 2016; Caglayan et al., 2018; Libovick´y and Helcl, 2017). Very recent work documents the use of visual information for in- terpreting implicit language (Collell …

Content truncated to 500 words