UoR at SemEval-2020 Task 4: Pre-trained Sentence Transformer Models for Commonsense Validation and Explanation

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 430–436 Barcelona, Spain (Online), December 12, 2020. 430 UoR at SemEval-2020 Task 4: Pre-trained Sentence Transformer Models for Commonsense Validation and Explanation Thanet Markchom, Bhuvana Dhruva, Chandresh Pravin, Huizhi Liang University of Reading White Knights, Berkshire, RG6 6AH United Kingdom t.markchom@pgr.reading.ac.uk, bhuvna 99@yahoo.co.uk, c.pravin@pgr.reading.ac.uk, huizhi.liang@reading.ac.uk Abstract SemEval Task 4 Commonsense Validation and Explanation Challenge is to validate whether a system can differentiate natural language statements that make sense from those that do not make sense. Two subtasks, A and B, are focused in this work, i.e., detecting against-common-sense statements and selecting explanations of why they are false from the given options. Intuitively, commonsense validation requires additional knowledge beyond the given statements. Therefore, we propose a system utilising pre-trained sentence transformer models based on BERT, RoBERTa and DistillBERT architectures to embed the statements before classification. According to the results, these embeddings can improve the performance of the typical MLP and LSTM classifiers as downstream models of both subtasks compared to regular tokenised statements. These embedded statements are shown to comprise additional information from external resources which help validate common sense in natural language. 1 Introduction Against-common-sense statements are statements which are contradictory, either partly or entirely, to the fact or do not make sense in reality. Understanding these statements can be challenging due to the variety and complexity of natural languages, yet it can be highly beneficial for many purposes such as detecting false information or validating statements generated by computer program. To explore this approach, Task4: Commonsense Validation and Explanation Challenge (ComVE) is introduced as a challenge in SemEval-2020. The main goal of ComVE is to develop a system that can differentiate and explain counterfactual natural language statements. This task consists of three subtasks, Subtask A, Subtask B and Subtask C. In Subtask A, two statements are given, one is against common sense while the other is not. The objective of Subtask A is to identify the statement that is contradictory to common sense. For Subtask B and C, the objectives are to explain why a given statement is contradict to the fact by using different methods. In Subtask B, an explanation is selected from the given options, while, an entire explanation has to be generated from scratch in Subtask C. The total detail can be found in (Wang et al., 2019; Wang et al., 2020) To decide whether a given statement makes sense, additional context may be required. In such case, a sentence embedding model becomes useful for including this additional information by semantically encoding statements before classification. In this work, we study on using pre-trained sentence transformer models (Reimers and Gurevych, 2019) which are state-of-the-art transformer models fine-tuned for sentence embedding to solve the problem in Subtask A and Subtask B. These models were modified based on three widely used transformer models with different configurations. In our approach, these pre-trained models are used to embed the statements so that the embedded statements can be classified more effectively to …

Content truncated to 500 words