Visuo-Linguistic Question Answering (VLQA) Challenge

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4606–4616 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4606 Visuo-Linguistic Question Answering (VLQA) Challenge Shailaja Keyur Sampat, Yezhou Yang and Chitta Baral Arizona State University Tempe, AZ, USA {ssampa17⇤, yz.yang, chitta}@asu.edu Abstract Understanding images and text together is an important aspect of cognition and building ad- vanced Artificial Intelligence (AI) systems. As a community, we have achieved good bench- marks over language and vision domains sep- arately, however joint reasoning is still a chal- lenge for state-of-the-art computer vision and natural language processing (NLP) systems. We propose a novel task to derive joint infer- ence about a given image-text modality and compile the Visuo-Linguistic Question An- swering (VLQA) challenge corpus in a ques- tion answering setting. Each dataset item con- sists of an image and a reading passage, where questions are designed to combine both vi- sual and textual information i.e., ignoring ei- ther modality would make the question unan- swerable. We first explore the best existing vision-language architectures to solve VLQA subsets and show that they are unable to rea- son well. We then develop a modular method with slightly better baseline performance, but it is still far behind human performance. We believe that VLQA will be a good benchmark for reasoning over a visuo-linguistic context. The dataset, code and leaderboard is available at https://shailaja183.github.io/vlqa/. 1 Introduction Question answering (QA) is a crucial way to eval- uate the system’s ability to understand text and images. In recent years, a large body of natural lan- guage QA (NLQA) datasets and visual QA (VQA) datasets have been compiled to evaluate the ability of a system to understand text and images. For most VQA datasets, the text is used merely as a question-answering mechanism rather than an ac- tual modality that provides contextual information. On the other hand, deriving inference from com- bined visual and textual information is an important ⇤corresponding author Figure 1: Example of Visuo-Linguistic Question An- swering (VLQA) task for joint reasoning over image- text context. skill for humans to perform day-to-day tasks. For example, product assembly using instruction man- uals, navigating roads while following street signs, interpreting visual representations (e.g., charts) in various documents such as newspapers and reports, understanding concepts using textbook-style learn- ing, etc. The importance of joint reasoning has also been emphasized in the design of standard- ized / psychometric tests like PISA (OECD, 2019) and GRE1, as evident from Figure 2. PISA as- sessments conducted post 2018 take into account “the evolving nature of reading in digital societies- which requires an ability to compare, contrast and integrate information from multiple sources”. The GRE has ‘data interpretation’ questions that assess a student’s ability to “analyze given data as a com- bination of text and charts.” Both the aforementioned evidence motivate the need to develop Visuo-Linguistic QA (VLQA) sys- tem, posing a further challenge to state-of-the-art 1https://www.oecd.org/pisa/, https://www.ets.org/gre/ 4607 Figure 2: Examples of joint-reasoning questions in standardized tests2(boldface represents correct answer) vision and …

Content truncated to 500 words