HR@JUST Team at SemEval-2020 Task 4: The Impact of RoBERTa Transformer for Evaluation Common Sense Understanding

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 521–526 Barcelona, Spain (Online), December 12, 2020. 521 HR@JUST team at SemEval-2020 Task 4: The impact of RoBERTa transformer for evaluation common sense understanding Heba Al-Jarrah hebaatta96@gmail.com Rahaf Al-Hammouri rahafalhamouri96@gmail.com Department of Computer Science Jordan University of Science and Technology Irbid, Jordan Mohammad AL-Smadi maalsmadi9@just.edu.jo Abstract This paper describes the results of our team HR@JUST participation at SemEval-2020 Task 4 -Commonsense Validation and Explanation (ComVE) for the POST evaluation period. The provided task consists of three sub-tasks, we participate in task A. We considered a state-of-the-art approach for solving this task by performing RoBERTa model with no Next Sentences Prediction (NSP), dynamic masking, larger training data, and larger batch size. The achieved results show that we got the 11th rank on the final test set leaderboard with an accuracy of 91.3%. 1 Introduction The concept of common sense in natural language systems has received increasing attention by researchers, as it is considered a long-term problem, and an important challenge in NLP. For example, , “He drinks milk”, is a logical sentence, while, ” He drinks apple”, is not a logic sentence. Looking at the example we know that it is self-evident that a human can distinguish between logical and illogical sentences due to common sense reasoning, while the trained model is weak in common sense and consider as a difficult task in NLP. by using language models trained on large corpora with neural networks (Trinh and Le, 2018; Peters et al., 2018) achieved a good result compared to previous models, but not satisfactory enough. using traditional machine learning algorithm (Ostermann et al., 2018) still have not had good results so far compared with human performance. Although these efforts, it is still unacceptable to rely on them for common sense problems. Natural language understanding is one of the main problems in NLP and the concept of common sense falls under this field. Where it is difficult to distinguish between the meanings of words in terms of semantic meaning and syntax structure. In the SemEval-2020 Task 4 subtask A, the organizers provide a dataset contains of 10 sentences: {s1, s2, o1, o2, o3, r1, r2, r3}. s1 and s2 are two sentences that are similar in syntactic and structure but different in a few words, that require the participants to build a model able to identify which one makes sense. SemEval is the International Workshop on Semantic Evaluation that has evolved from SemEval.This workshop aims to evaluate semantic analysis systems where is poses some problems related to natural languages (e.g., lexical semantics, Common Sense Knowledge, Knowledge Extraction, Humour, Emphasis, and Sentiment), the SemEval-2020 being the 14th workshop on semantic evaluation. Task 4 (Wang et al., 2020) present three sub-tasks with annotated datasets in the English language. The task for participating teams is to distinguish the sentence that common sense from others. The dataset contains two sentences s1 and s2 are similar in syntactic and structure but different in a few words, that …

Content truncated to 500 words