Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10158 - 10166 December 7-11, 2022 ©2022 Association for Computational Linguistics Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering Deepanway Ghosal DeCLaRe Navonil Majumder DeCLaRe Rada Mihalcea Soujanya Poria DeCLaRe DeCLaReDeCLaRe Lab, Singapore University of Technology and Design, Singapore University of Michigan, USA {deepanway_ghosal@mymail.,navonil_majumder@,sporia@}sutd.edu.sg mihalcea@umich.edu Abstract We propose a simple refactoring of multi- choice question answering (MCQA) tasks as a series of binary classifications. The MCQA task is generally performed by scoring each (question, answer) pair normalized over all the pairs, and then selecting the answer from the pair that yield the highest score. For n answer choices, this is equivalent to an n-class clas- sification setup where only one class (true an- swer) is correct. We instead show that clas- sifying (question, true answer) as positive in- stances and (question, false answer) as neg- ative instances is significantly more effective across various models and datasets. We show the efficacy of our proposed approach in differ- ent tasks – abductive reasoning, commonsense question answering, science question answer- ing, and sentence completion. Our DeBERTa binary classification model reaches the top or close to the top performance on public leader- boards for these tasks. The source code of the proposed approach is available at https: //github.com/declare-lab/TEAM. 1 Introduction Starting with the early Text Retrieval Conference (TREC) community-wide evaluations of textual question answering (Voorhees et al., 1999), all the way to the recent work on multimodal question answering (Lei et al., 2018; Tapaswi et al., 2016; Jang et al., 2017; Castro et al., 2020) and common- sense question answering (Sap et al., 2019; Talmor et al., 2019), the task has become a staple of the natural language processing research community. One of the major challenges encountered in ques- tion answering is the evaluation, which often re- quires human input to evaluate the textual answers thoroughly. Because of this, the alternative that has been proposed is that of multi-choice question answering, where the correct answer is provided together with other incorrect answers. The task is thus transformed into that of answer classifica- tion, where a system has to select one answer from the choices provided. While there are drawbacks associated with this evaluation metric, it has been widely adopted because of its benefit of providing a clear evaluation methodology. In this paper, we reformulate the task of multi-choice question answering as a binary classification task and show that this re-framing leads to significant performance improvements on several datasets. Importantly, this formulation brings flexibility to the overall question-answering setup, as it reduces the dependence on the up-front availability of multiple candidate answers. Using our method – TEAM (Two is bEtter thAn Many), candidate answers can be produced and evaluated for correctness on the fly, and thus the answer classification component can be also used in conjunction with more natural settings that use open-ended answer generation (Castro et al., 2022; Sadhu et al., 2021). 2 Methodology …

Content truncated to 500 words