SemEval-2020 Task 4: Commonsense Validation and Explanation

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 307–321 Barcelona, Spain (Online), December 12, 2020. 307 SemEval-2020 Task 4: Commonsense Validation and Explanation Cunxiang Wang1,2*, Shuailong Liang3*, Yili Jin4, Yilong Wang1, Xiaodan Zhu5 and Yue Zhang2 1Zhejiang University, China; 2School of Engineering, Westlake University, China 3Singapore University of Technology and Design, Singapore; 4Sun Yat-Sen University, China; 5Queen’s University, Canada wangcunxiang, zhangyue@westlake.edu.cn, shuailong liang@mymail.sutd.edu.sg, zhu2048@gmail.com Abstract In this paper, we present SemEval-2020 Task 4, Commonsense Validation and Explanation (ComVE), which includes three subtasks, aiming to evaluate whether a system can distinguish a natural language statement that makes sense to humans from one that does not, and provide the reasons. Specifically, in our first subtask, the participating systems are required to choose from two natural language statements of similar wording the one that makes sense and the one does not. The second subtask additionally asks a system to select the key reason from three options why a given statement does not make sense. In the third subtask, a participating system needs to generate the reason. We finally attracted 39 teams participating at least one of the three subtasks. For Subtask A and Subtask B, the performances of top-ranked systems are close to that of humans. However, for Subtask C, there is still a relatively large gap between systems and human performance. The dataset used in our task can be found at https://github.com/wangcunxiang/SemEval2020- Task4-Commonsense-Validation-and-Explanation; The leaderboard can be found at https://competitions.codalab.org/competitions/21080#results. 1 Introduction In the past decades, computer’ ability in processing natural language has significantly improved. However, its intelligence for understanding common sense expressed in language is still limited. For example, it is straightforward for humans to judge that the following sentence is plausible, or makes sense: “John put a turkey into a fridge” while “John put an elephant into the fridge” does not, but it is non-trivial for a computer to tell the difference. Arguably, commonsense reasoning plays a central role in a natural language understanding system (Davis, 2017). It is essential to gauge how well computers can understand whether a given statement makes sense. In our task, we take an operational definition of making sense by asking human subjects to generate natural language statements that obey or violate their commonsense knowledge about the world. 1 Many existing tasks embed the evaluation of commonsense understanding in other tasks such as co- reference resolution (Levesque et al., 2012; Morgenstern and Ortiz, 2015), subsequent event prediction (Roemmele et al., 2011), ordinal common-sense inference (Zhang et al., 2017), situations with adver- sarial generations (Zellers et al., 2018), event validation (Wang et al., 2018), reading comprehension (Mostafazadeh et al., 2016; Ostermann et al., 2018b; Ostermann et al., 2018a), dialogue (Cui et al., 2020) and QA (Davis, 2016; Talmor et al., 2018; Mihaylov et al., 2018). They verify whether a system is equipped with common sense by testing whether the system can give a correct answer when the input does not contain such knowledge. The above tasks do not directly evaluate commonsense validation and they …

Content truncated to 500 words