Samsung Research Poland (SRPOL) at SemEval-2022 Task 9: Hybrid Question Answering Using Semantic Roles

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1263 - 1273 July 14-15, 2022 ©2022 Association for Computational Linguistics Samsung Research Poland (SRPOL) at SemEval-2022 Task 9: Hybrid Question Answering Using Semantic Roles Tomasz Dryja´nski1, Monika Zaleska1, Bartłomiej Ku´zma1, Artur Bła˙zejewski1, Zuzanna Bordzicka1, Klaudia Firl ˛ag1, Christian Goltz1, Maciej Grabowski1, Jakub Jo´nczyk1, Grzegorz Kłosi´nski1, Natalia Paszkiewicz1, Bartłomiej Paziewski1, Jarosław Piersa1, Paweł Bujnowski1, Piotr Andruszkiewicz1,2 {t.dryjanski; m.zaleska; b.kuzma; a.blazejewsk; z.bordzicka; k.firlag; c.goltz; m.grabowski2; j.jonczyk; g.klosinski; n.paszkiewic; b.paziewski; j.piersa; p.bujnowski; p.andruszki2}@samsung.com 1 Samsung Research Poland, Warsaw 2 Warsaw University of Technology, Warsaw Abstract In this work we present an overview of our winning system for the R2VQ — Competence- based Multimodal Question Answering task, with the final exact match score of 92.53%. The task is structured as question-answer pairs, querying how well a system is capable of competence-based comprehension of recipes. We propose a hybrid of a rule-based system, Question Answering Transformer, and a neu- ral classifier for N/A answers recognition. The rule-based system focuses on intent identifica- tion, data extraction and response generation. 1 Introduction The goal of the task1 was to develop a system apply- ing existing knowledge to new situations, demon- strating a kind of understanding of a real-world do- main. The competition presents a QA2 challenge re- quiring linguistic and cognitive competencies that humans have while speaking and reasoning (Tu et al., 2022). The task dataset contains questions belonging to "question families" based on CLEVR (Johnson et al., 2016), reflecting specific reasoning compe- tences. These families were explicitly marked as 19 categories, the last one having no answer (N/A), but direct reference to these categories was prohib- ited by the task requirements. The cooking recipes included in the dataset were provided with exceptionally extensive annotations containing semantic information. The authors ap- plied CRL and span-based SRL using VerbAtlas 1https://competitions.codalab.org/ competitions/34056 (access Apr 28th, 2022). 2Abbreviations used in the text: QA: Question Answering; SRL: Semantic Role Labeling (or Labels); CRL: Cooking Role Labeling; EM: Exact Match; RC: Reading Comprehen- sion; DNN: Deep Neural Networks. (Di Fabio et al., 2019) for the reference inventory of frames and semantic roles. Subsequently, hu- man annotators were asked to validate and correct frames and argument labels. The dataset was split into training (26,526), val- idation (3,829) and test set (3,442 questions). At the competition evaluation stage, the answers to the latter were not revealed, but the annotations were retained. Our source code is available on GitHub3. 2 Related Work In the recent years, deep learning systems trained on large datasets began to outperform humans and other algorithms in the whole QA discipline. Chal- lenges presented in works such as SQuAD (Ra- jpurkar et al., 2018), MS MARCO (Nguyen et al., 2016), CoQA (Reddy et al., 2019), multilingual MLQA (Lewis et al., 2020) and others popularized various machine learning models for extractive QA. Meanwhile, visual and multimodal QA contests started to appear, e.g. VQA (Antol et al., 2015) or Audio-Visual Scene-Aware Dialog (Alamri et al., 2019). They require understanding of images, …

Content truncated to 500 words