ANA at SemEval-2020 Task 4: MUlti-task learNIng for cOmmonsense reasoNing (UNION)

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 367–373 Barcelona, Spain (Online), December 12, 2020. 367 ANA∗ at SemEval-2020 Task 4: mUlti-task learNIng for cOmmonsense reasoNing (UNION) Anandh Perumal, Chenyang Huang, Amine Trabelsi, Osmar R. Za¨ıane Alberta Machine Intelligence Institute, University of Alberta {anandhpe, chenyangh, atrabels,zaiane}@ualberta.ca Abstract In this paper, we describe our mUlti-task learNIng for cOmmonsense reasoNing (UNION) system submitted for Task C of the SemEval2020 Task 4, which is to generate a reason explaining why a given false statement is non-sensical. However, we found in the early experiments that simple adaptations such as fine-tuning GPT2 often yield dull and non-informative generations (e.g. simple negations). In order to generate more meaningful explanations, we propose UNION, a unified end-to-end framework, to utilize several existing commonsense datasets so that it allows a model to learn more dynamics under the scope of commonsense reasoning. In order to perform model selection efficiently, accurately and promptly, we also propose a couple of auxiliary automatic evaluation metrics so that we can extensively compare the models from different perspectives. Our submitted system not only results in a good performance in the proposed metrics but also outperforms its competitors with the highest achieved score of 2.10 for human evaluation while remaining a BLEU score of 15.7. Our code is made publicly availabled 1. 1 Introduction Common sense reasoning is one of the long-standing problems in natural language understanding. Previous work on modeling common sense knowledge deals mainly with indirect tasks such as co- reference resolution (Sakaguchi et al., 2019), or selecting the plausible situation based on the given subject or scenario (Zellers et al., 2018). In this paper, we present our system that we devised to tackle Task C, Explanation (Generation), of the SemEval 2020 Task 4 - Commonsense Validation and Explanation (ComVE). Given a false or non-sensical statement, the task consists of generating the reason why the given statement does not make sense. We propose a mUlti-task learNIng for cOmmonsense reasoNing (UNION). It combines datasets including ComVE (Wang et al., 2019), OpenBook (Mihaylov et al., 2018), Common sense Explanation (CoS-E) (Rajani et al., 2019) and Open Mind Common Sense (OMCS) (Singh et al., 2002) in a multi-task framework. The backbone of UNION is a large pre-trained language model – GPT2. We compare the proposed system to different baselines and report a significant improvement in BLEU score and other evaluation metrics. Our proposed model achieves a human evaluation score of 2.10, which ranked first on the final leader board for Task C of SemEval 2020 Task 4. In our initial submission, we used ComVE, CoS-E, and OpenBook datasets for training. In that case, a BLEU score of 15.7 is achieved. Pretraining the model with OMCS dataset further improved the BLEU score by 0.7. In addition, we show some of our generations in the appendix. 2 Background and Related Work The common sense reasoning in machines has been one of the most challenging problems, and a major critical missing component of Artificial Intelligence (AI). It could …

Content truncated to 500 words