A Semantic-based Method for Unsupervised Commonsense Question Answering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3037–3049 August 1–6, 2021. ©2021 Association for Computational Linguistics 3037 A Semantic-based Method for Unsupervised Commonsense Question Answering Yilin Niu1∗, Fei Huang1∗, Jiaming Liang1, Wenkai Chen2, Xiaoyan Zhu1, Minlie Huang1† 1 The CoAI group, DCST; 1 Institute for Artificial Intelligence; 1 State Key Lab of Intelligent Technology and Systems; 1 Beijing National Research Center for Information Science and Technology; 1 Tsinghua University, Beijing 100084, China. 2 School of Computer Science and Technology, Beijing University of Posts and Telecommunications niuyl14@tsinghua.org.cn {f-huang,liangjm18}18@mails.tsinghua.edu.cn wkchen630@gmail.com {zxy-dcs,aihuang}@tsinghua.edu.cn Abstract Unsupervised commonsense question answer- ing is appealing since it does not rely on any labeled task data. Among existing work, a popular solution is to use pre-trained language models to score candidate choices directly con- ditioned on the question or context. However, such scores from language models can be eas- ily affected by irrelevant factors, such as word frequencies, sentence structures, etc. These distracting factors may not only mislead the model to choose a wrong answer but also make it oversensitive to lexical perturbations in can- didate answers. In this paper, we present a novel SEmantic- based Question Answering method (SEQA) for unsupervised commonsense question an- swering. Instead of directly scoring each an- swer choice, our method first generates a set of plausible answers with generative models (e.g., GPT-2), and then uses these plausible an- swers to select the correct choice by consider- ing the semantic similarity between each plau- sible answer and each choice. We devise a sim- ple, yet sound formalism for this idea and ver- ify its effectiveness and robustness with exten- sive experiments. We evaluate the proposed method on four benchmark datasets, and our method achieves the best results in unsuper- vised settings. Moreover, when attacked by TextFooler (Jin et al., 2020) with synonym re- placement, SEQA demonstrates much less per- formance drops than baselines, thereby indicat- ing stronger robustness. 1 Introduction Pre-trained language models have been widely used for commonsense question answering. Finetuning pre-trained models on task-specific data produces many state-of-the-art results (Wang et al., 2020; *Equal contribution †Corresponding author: Minlie Huang. Figure 1: Two examples of commonsense question an- swering, where the baseline (Pro-A) is oversensitive to lexical perturbations (SR for synonym replacement and ST for sentence structure transformation). The scores from Pro-A and our method for each answer choice are shown in the right columns. The underlined score indi- cates the answer choice selected by a method. Khashabi et al., 2020; Lin et al., 2019). However, this requires amounts of labeled task data. There- fore, it is vital to study unsupervised commonsense question answering without relying on any labeled downstream task data. In this paper, we investigate multiple-choice commonsense question answering tasks in an unsupervised setting: given a question and a set of answer choices, a model is required to predict the most reasonable answer choice for the question, but without access to any labeled task data. …

Content truncated to 500 words