CUHK at SemEval-2020 Task 4: CommonSense Explanation, Reasoning and Prediction with Multi-task Learning

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 391–400 Barcelona, Spain (Online), December 12, 2020. 391 CUHK at SemEval-2020 Task 4: CommonSense Explanation, Reasoning and Prediction with Multi-task Learning Hongru Wang1, Xiangru Tang3, Sunny Lai1, Kwong Sak Leung*1, Jia Zhu4, Gabriel Pui Cheong Fung2, Kam-Fai Wong2 1Department of Computer Science and Engineering The Chinese University of Hong Kong 2 Department of Systems Engineering and Engineering Management The Chinese University of Hong Kong 3Institute of Computing Technology, Chinese Academy of Sciences 4School of Computer Science, South China Normal University {hrwang, slai, ksleung}@cse.cuhk.edu.hk jzhu@m.scnu.edu.cn {pcfung, kfwong}@se.cuhk.edu.hk Abstract This paper describes our system submitted to task 4 of SemEval 2020: Commonsense Validation and Explanation (ComVE) which consists of three sub-tasks. The challenge is to directly validate whether the system can recognize natural language statements that make sense from those that do not, and also require to generate reasonable explanation. Based on BERT architecture with multi-task setting, we propose an effective and interpretable “Explain, Reason and Predict” (ERP) system to solve the three sub-tasks about commonsense: (a) Validation, and (c) Explanation, (b) Reasoning, following the order of the competition. Inspired by cognitive studies of common sense, our system first generate a reason or understanding of the sentences and then choose which one statement makes sense, which is achieved by multi-task learning. The rational experiment validates our assumption and boost the performance. During the post-evaluation, our system has reached 92.9% accuracy in subtask A (rank 11), 89.7% accuracy in subtask B (rank 8), and BLEU score of 12.9 in subtask C (rank 9)1. 1 Introduction How to integrate common sense into natural language models is attracting more and more attention. Common sense, as ordinarily conceived, present themselves as the aspect of the grammar of expressions and sentences on which their semantic properties and relations depend (Asher and Vieu, 1995). And a critical difference of text understanding between humans and machines lies in the fact that humans can access commonsense knowledge while processing text, which helps them draw inferences about effects that are not mentioned in a paragraph. Therefore, it’s a fundamental question on how to validate whether a system has a commonsense capability, and more importantly, let the system explain how it reasons using hidden facts. Existing benchmarks measure commonsense knowledge indirectly and without explanation. Also, existing datasets test common sense indirectly through tasks that require extra knowledge, such as co-reference resolution, or reading comprehension. They verify whether a system is equipped with common sense by testing whether it can give a correct answer to make the complete sentence reasonable. However, there are some restrictions on such benchmarks. First, they do not provide a straight quantitatively standard to measure sense masking capability. Second, they do not explicitly identify the key factors required in a sense-making process. And also, they do not need the model to explain why it makes that prediction. Common sense reasoning require the agent or the model to utilize a world knowledge to take inferences or deep semantic understanding, not …

Content truncated to 500 words