NLP@JUST at SemEval-2020 Task 4: Ensemble Technique for BERT and Roberta to Evaluate Commonsense Validation

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 574–579 Barcelona, Spain (Online), December 12, 2020. 574 NLP@JUST at SemEval-2020 Task 4: Ensemble Technique for BERT and Roberta to Evaluate Commonsense Validation Emran Al Bashabsheh Computer Science Jordan University of Science and Technology Irbid, Jordan emranalbashabsheh@gmail.com Ayah Abu Aqouleh Computer Science Jordan University of Science and Technology Irbid, Jordan ayaalkhader96@gmail.com Mohammad AL-Smadi Computer Science Jordan University of Science and Technology Irbid, Jordan masmadi@just.edu.jo Abstract This paper presents the work of the NLP@JUST team at SemEval-2020 Task 4 competition that related to commonsense validation and explanation (ComVE) task. The team participates in sub-taskA (Validation) which related to validation that checks if the text is against common sense or not. Several models have trained (i.e. Bert, XLNet, and Roberta), however, the main models used are the RoBERTa-large and BERT Whole word masking. As well as, we utilized the results from both models to generate final prediction by using the average Ensemble technique, that used to improve the overall performance. The evaluation result shows that the implemented model achieved an accuracy of 93.9% obtained and published at the post-evaluation result on the leaderboard. 1 Introduction Recently, Natural Language Processing (NLP) has got more attention because it is able to deal with human and machine communication without human intervention (Lytinen, 2005). NLP is a subset of Artificial Intelligence (AI) that grants the machines ability to read and understand human words using Machine Learning (ML) and Deep Learning (DL) models. However, it is used to solve a large scope of tasks such as Machine Translation, Speech Recognition, Sentiment Analysis, etc. NLP has two subsections Natural Language Understanding (NLU) and Natural Language Generation (NLG). Furthermore, NLU with commonsense has gained the researcher’s interests in the last few years(Wang et al., 2019). Commonsense is known as the knowledge that captured about the world’s continents and the implicit reasoning process (Storks et al., 2019). As known the human have obtained commonsense knowledge from their daily life, they can distinguish the sentence is against commonsense or not from their previous knowledge. The Commonsense task is considered as one of the important tasks of NLU in NLP that makes the machines more intelligent (Storks et al., 2019). Hence, there are many works in the literature focus on commonsense tasks, and various benchmarks created to evaluate a trained model’s capabilities to capture the commonsense validation task. For example, (Wang et al., 2019) introduced a new benchmark that has two tasks which are: the Sen-Making task and Explanation task, to evaluate the system can judge the text makes sense or not and provide reasoning why the text is not commonsense (Wang et al., 2019). This paper presenting NLP@JUST teamwork submitted at SemEval-2020 Task 4 competition at a post-evaluation time. This task is related to commonsense validation and reasoning, released by (Wang et al., 2020). SemEval-2020 Task 4 has three subtasks: Validation, Explanation (Multi-Choice), and Explanation (Generation). Specifically, this paper focuses on the commonsense validation task. Mainly the RoBERTa (Liu et al., …

Content truncated to 500 words