UI at SemEval-2020 Task 4: Commonsense Validation and Explanation by Exploiting Contradiction

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 614–619 Barcelona, Spain (Online), December 12, 2020. 614 CSUI at SemEval-2020 Task 4: Commonsense Validation and Explanation by Exploiting Contradiction Kerenza Doxolodeo, Rahmad Mahendra Faculty of Computer Science, Universitas Indonesia Depok, West Java, Indonesia kerenza.doxolodeo@ui.ac.id, rahmad.mahendra@cs.ui.ac.id Abstract This paper describes our submissions into the ComVe challenge, the SemEval 2020 Task 4. This evaluation task consists of three sub-tasks that test commonsense comprehension by identifying sentences that do not make sense and explain why they do not. In subtask A, we use Roberta to find which sentence does not make sense. In subtask B, besides using BERT, we also experiment with replacing the dataset with MNLI when selecting the best explanation from the provided op- tions why the given sentence does not make sense. In subtask C, we utilize the MNLI model from subtask B to evaluate the explanation generated by Roberta and GPT-2 by exploiting the contra- diction of the sentence and their explanation. Our system submission records a performance of 88.2%, 80.5%, and BLEU 5.5 for those three subtasks, respectively. 1 Introduction Natural Language Understanding (NLU) has seen increasing attention in recent times. It requires not only the capability to deduce the semantics of the text but also commonsense knowledge of the world. One of the prominent tasks to test how well a machine recognizes and understands the human language is Natural Language Inference, in which given the pair of texts (a hypothesis and a premise), and the model deduces whether the premise entails or contradicts the hypothesis. Winograd Scheme Challenge (Morgenstern et al., 2016) is another task that provides a text and a question regarding the text with two possible answers, and the model must select the correct one. However, the question is designed that both answers are possible, grammatically wise, and the only way to answer is to have access to the implicit commonsense knowledge. While the aforementioned tasks are the application of commonsense, SemEval 2020 Task 4: Com- monsense Validation and Explanation (Wang et al., 2020) puts emphasis purely on the commonsense itself. In this paper, we present our submission system that relies on the Transformers-based model (De- vlin et al., 2019). On the other hand, one of the biggest concerns of the ComVe subtask is its small size. We know that the explanatory NLU model is prone to over-fitting and may generate inconsistent expla- nation (Camburu et al., 2020). A potential solution is to augment the dataset by leveraging the existing commonsense knowledge. 2 Task Description SemEval 2020 Task 4: Commonsense Validation and Explanation (ComVe) challenges three subtasks: identifying which member of a pair of a similar sentence does not make sense, selecting the best expla- nation why a sentence does not make sense, and generate an explanation from scratch why a sentence does not make sense. The dataset for ComVe task consists of 10K instances in train set and 1K instances in test set This work is licensed under a Creative Commons Attribution 4.0 …

Content truncated to 500 words