HIT&QMUL at SemEval-2022 Task 9: Label-Enclosed Generative Question Answering (LEG-QA)

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1256 - 1262 July 14-15, 2022 ©2022 Association for Computational Linguistics HIT&QMUL at SemEval-2022 Task 9: Label-Enclosed Generative Question Answering (LEG-QA) Weihe Zhai*1,2, Mingqiang Feng*1, Arkaitz Zubiaga2, Bingquan Liu†1 1Harbin Institute of Technology, China 2Queen Mary University of London, UK {weihezhai, mqfeng}@insun.hit.edu.cn a.zubiaga@qmul.ac.uk liubq@hit.edu.cn Abstract This paper presents the second place system for the R2VQ: competence-based multimodal question answering shared task. The task con- sisted in building question answering systems that could process procedural recipes involv- ing both text and image, and enriched with semantic and cooking roles. We tackled the task by using a text-to-text generative model based on the transformer architecture, with the aim of generalising across different question types. Our proposed architecture incorporates a novel approach for enriching input texts by incorporating semantic and cooking role la- bels through what we call Label-Enclosed Gen- erative Question Answering (LEG-QA). Our model achieves a score of 91.3, with a signifi- cant improvement over the baseline (65.34) and close to the top-ranked system ((92.5). After describing the submitted system, we analyse the impact of the different components of LEG- QA as well as perform an error analysis. 1 Introduction The objective of text-and-image multimodal ques- tion answering (QA) is to jointly leverage both textual and visual information to mutually inform each other for semantic reasoning (Ben-Younes et al., 2017). A SemEval 2022 shared task titled Competence-based Multimodal Question Answer- ing (R2VQ) focuses on this task. In the R2VQ task, participants were invited to develop QA mod- els to resolve questions associated with procedural recipe instructions. The corpus provided for the task is made of recipes, which include rich seman- tic annotations and where textual instructions are aligned with images illustrating them. The authors proposed to address a set of 18 question families, for which participants could develop and evaluate their own proposed solutions. *contribute equally †corresponding author This paper describes the participation of the HIT&QMUL team in the R2VQ task, for which we proposed a methodology that we call label- enclosed generative question answering (LEG- QA). Through this methodology, we proposed en- closing labels providing semantic information em- bedded in the input texts. This methodology has proven competitive by achieving a score of 91.3% in exact match accuracy, ranking 2nd overall in the competition. Our code is available at https: //github.com/weihezhai/ HIT-QMUL-at-SemEval-2022-Task-9. 2 Task and System Description Building on the transformer architecture (Vaswani et al., 2017), we use T5 an encoder-decoder model (Raffel et al., 2020) implemented using Hugging Face1. We chose T5 given its reasonably good gen- eral language learning abilities, and provided that the downstream task of R2VQ covers a diverse set of task types that are also shared by T5. Intuitively, a task-agnostic model like T5 would be expected to perform well on R2VQ. We further adapt the T5 model to the task by altering the input to include semantic and cooking role labels enclosed in the textual recipe instructions. 2.1 Task and Data The R2VQ …

Content truncated to 500 words