Explanations for CommonsenseQA: New Dataset and Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3050–3065 August 1–6, 2021. ©2021 Association for Computational Linguistics 3050 Explanations for CommonsenseQA: New Dataset and Models Shourya Aggarwal1, Divyanshu Mandowara1, Vishwajeet Agrawal1, Dinesh Khandelwal2, Parag Singla1, Dinesh Garg2 1IIT Delhi, 2IBM Research AI, India aggarwal.shourya@gmail.com, divyanshu.mandowara92@gmail.com vishwa.grawal@gmail.com, dikhand1@in.ibm.com parags@cse.iitd.ac.in, garg.dinesh@in.ibm.com Abstract CommonsenseQA (CQA) (Talmor et al., 2019) dataset was recently released to advance the research on common-sense question an- swering (QA) task. Whereas the prior work has mostly focused on proposing QA mod- els for this dataset, our aim is to retrieve as well as generate explanation for a given (ques- tion, correct answer choice, incorrect answer choices) tuple from this dataset. Our expla- nation definition is based on certain desider- ata, and translates an explanation into a set of positive and negative common-sense prop- erties (aka facts) which not only explain the correct answer choice but also refute the in- correct ones. We human-annotate a first-of- its-kind dataset (called ECQA) of positive and negative properties, as well as free-flow ex- planations, for 11K QA pairs taken from the CQA dataset. We propose a latent representa- tion based property retrieval model as well as a GPT-2 based property generation model with a novel two step fine-tuning procedure. We also propose a free-flow explanation genera- tion model. Extensive experiments show that our retrieval model beats BM25 baseline by a relative gain of 100% in F1 score, property generation model achieves a respectable F1 score of 36.4, and free-flow generation model achieves a similarity score of 61.9, where last two scores are based on a human correlated se- mantic similarity metric. 1 Introduction The field of automated question answering (QA) has witnessed a rapid progress in the past few years, sometimes beating even human perfor- mance (Zhang et al., 2020). The reasons behind this trend include (i) emergence of large-sized QA datasets such as SQuAD (Ra- jpurkar et al., 2016), HotpotQA (Yang et al., 2018), CommonsenseQA (Talmor et al., 2019), NaturalQA (Kwiatkowski et al., 2019), etc., and (ii) emergence of powerful, large scale, pre- Question: Where is a frisbee in play likely to be? Answer Choices: outside park roof tree air Our Explanation: Positives Properties 1) A frisbee is a concave plastic disc designed for skimming through the air as an outdoor game. Negative Properties 1) A frisbee can be outside anytime, even while not in play. 2) A frisbee can be in a park anytime, even while not in play. 3) A frisbee can be on a roof after play. 4) A frisbee can be in a tree after play. Free-Flow (FF) Explanation A frisbee is a concave plastic disc designed for skimming through the air as an outdoor game, so while in play it is most likely to be in the air. A frisbee can be outside or in a park anytime, and other options are possible only after play. CoS Explanation (Rajani et al., 2019): …

Content truncated to 500 words