SemEval-2022 Task 9: R2VQ – Competence-based Multimodal Question Answering

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1244 - 1255 July 14-15, 2022 ©2022 Association for Computational Linguistics SemEval-2022 Task 9: R2VQ – Competence-based Multimodal Question Answering Jingxuan Tu, Eben Holderness, Kyeongmin Rim, Kelley Lynch, Richard Brutti, James Pustejovsky Lab for Linguistics & Computation Department of Computer Science Brandeis University {jxtu,egh,krim,kmlynch, brutti,jamesp}@brandeis.edu Marco Maru1, Simone Conia1, Roberto Navigli2 Sapienza NLP Group 1Department of Computer Science 2Department of Computer, Control and Management Engineering Sapienza University of Rome first.lastname@uniroma1.it Abstract In this task, we identify a challenge that is re- flective of linguistic and cognitive competen- cies that humans have when speaking and rea- soning. Particularly, given the intuition that textual and visual information mutually inform each other for semantic reasoning, we formu- late a Competence-based Question Answering challenge, designed to involve rich semantic annotation and aligned text-video objects. The task is to answer questions from a collection of English language cooking recipes and videos, where each question belongs to a “question fam- ily” reflecting a specific reasoning competence. The data and task result is publicly available. 1 1 Introduction One of the fundamental goals of Artificial Intel- ligence (AI) has been to create systems that in- teract with human users fluently and intelligently, by demonstrating inferencing and reasoning capa- bilities that would be expected of a human part- ner. This includes a growing interest in posing larger challenges to end-to-end systems employing architectures with deep neural networks (DNNs) (Ribeiro et al., 2020; Prabhumoye et al., 2020; Rogers et al., 2021; Minaee et al., 2021). Here we argue that we should start focusing on linguistic competencies, and not just on Question Answer- ing (QA) skills or “challenge checklisting”. There are some moves in this direction already (John- son et al., 2017), but there is still no generally ac- cepted distinction in current Natural Language Pro- cessing (NLP) between challenge-based tasks and competence-based performance (Bentivogli et al., 2017). Analogous to human cognitive competen- cies, there is both a methodological and modeling advantage to focusing a system’s performance on 1https://competitions.codalab.org/ competitions/34056 competence-based learning rather than a narrowly defined task or challenge checklist. First we define competence-based knowledge, and then the questions that can be generated from such knowledge. While Chomsky (1965)’s dis- tinction between competence and performance has long been debated in linguistics, the term competence-based has been applied to a number of different concepts in both the science of learn- ing and educational communities (Bechtel et al., 1999; Voorhees, 2001; Chyung et al., 2006; Platan- ios et al., 2019; Hsiao et al., 2020). The common core to both is a concept capturing a coherent set of abilities that an individual has in a specific domain (Doignon and Falmagne, 1985; Heller et al., 2013). Here we focus on lexical competence as de- ployed in both single and multiple sentence com- position (Pustejovsky, 1995; Marconi, 1997; Geer- aerts, 2009; Asher, 2011). A competence-based question will query competence-based knowledge structures. For this task, lexical competence will involve the following: • Understanding implicit …

Content truncated to 500 words