JUSTers at SemEval-2020 Task 4: Evaluating Transformer Models against Commonsense Validation and Explanation

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 535–542 Barcelona, Spain (Online), December 12, 2020. 535 JUSTers at SemEval-2020 Task 4: Evaluating Transformer Models Against Commonsense Validation and Explanation Ali Fadel Jordan University of Science & Tech Irbid, Jordan aliosm1997@gmail.com Mahmoud Al-Ayyoub Jordan University of Science & Tech Irbid, Jordan maalshbool@just.edu.jo Erik Cambria Nanyang Technological University, Singapore cambria@ntu.edu.sg Abstract In this paper, we describe our team’s (JUSTers) effort in the Commonsense Validation and Explana- tion (ComVE) task, which is part of SemEval2020. We evaluate five pre-trained Transformer-based language models with various sizes against the three proposed subtasks. For the first two subtasks, the best accuracy levels achieved by our models are 92.90% and 92.30%, respectively, placing our team in the 12th and 9th places, respectively. As for the last subtask, our models reach 16.10 BLEU score and 1.94 human evaluation score placing our team in the 5th and 3rd places according to these two metrics, respectively. The latter is only 0.16 away from the 1st place human evaluation score. 1 Introduction Addressing the issue of commonsense understanding using deep learning algorithms and models has attracted increasing attention from the research community. Building a natural language processing (NLP) system that can understand both explicit and implicit knowledge, validate it and correct it before extracting important information is a task that is both hard and complex. At the same time, this task is essential to a large number of tasks like language modeling, word sense disambiguation and sentiment analysis (Cambria et al., 2009). Approaches like KnowBERT (Peters et al., 2019) leverage knowledge bases like WordNet (Miller et al., 1990) and DBPedia. Such knowledge bases provide a rich source of high quality, human-curated knowledge to fine-tune the internal hidden states of large language models like BERT (Devlin et al., 2018). Other approaches leverage external knowledge to enhance classification, e.g., Sentic LSTM (Ma et al., 2018) tackled the problem of targeted aspect-based sentiment analysis by embedding the knowledge from SenticNet (Cambria et al., 2020) into an attentive long short-term memory (LSTM) network. The Commonsense Validation and Explanation (ComVE) task (Wang et al., 2020) at SemEval2020 was proposed based on a dataset built by Wang et al. (Wang et al., 2019) to evaluate deep learning algorithms and models against commonsense tasks. The general purpose of the dataset is to test whether an NLP system can differentiate statements that make sense from those that do not. It consists of three subtasks: • Task A: Validation (Sentence Classification) In this subtask, the system is given two natural language statements with similar wordings and it is expected to distinguish the statement that makes sense from the one that does not. For example: Statement 1: He put a turkey into the fridge. correct Statement 2: He put an elephant into the fridge. • Task B: Explanation (Multiple Choice) In this subtask, a natural language statement that contradicts commonsense is given along with three reasons why it does not make sense. The system is expected to choose the …

Content truncated to 500 words