CS-NLP Team at SemEval-2020 Task 4: Evaluation of State-of-the-art NLP Deep Learning Architectures on Commonsense Reasoning Task

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 507–515 Barcelona, Spain (Online), December 12, 2020. 507 CS-NLP team at SemEval-2020 Task 4: Evaluation of State-of-the-art NLP Deep Learning Architectures on Commonsense Reasoning Task Sirwe Saeedi Western Michigan University, USA sirwe.saeedi@wmich.edu Aliakbar Panahi Virginia Commonwealth University, USA panahia@vcu.edu Seyran Saeedi Virginia Commonwealth University, USA saeedis@vcu.edu Alvis C Fong Western Michigan University, USA alvis.fong@wmich.edu Abstract In this paper, we investigate a commonsense inference task that unifies natural language un- derstanding and commonsense reasoning. We describe our attempt at SemEval-2020 Task 4 competition: Commonsense Validation and Explanation (ComVE) challenge. We discuss several state-of-the-art deep learning architectures for this challenge. Our system uses prepared labeled textual datasets that were manually curated for three different natural language inference subtasks. The goal of the first subtask is to test whether a model can distinguish between natural language statements that make sense and those that do not make sense. We compare the performance of several language models and fine-tuned classifiers. Then, we propose a method inspired by question/answering tasks to treat a classification problem as a multiple choice question task to boost the performance of our experimental results (96.06%), which is significantly better than the baseline. For the second subtask, which is to select the reason why a statement does not make sense, we stand within the first six teams (93.7%) among 27 participants with very competitive results. Our result for last subtask of generating reason against the nonsense statement shows many potentials for future researches as we applied the most powerful generative model of language (GPT-2) with 6.1732 BLEU score among first four teams1. Keywords: Artificial Intelligence, Natural Language Processing, Commonsense Reasoning and Knowl- edge, Language Models, Transformers, Self-Attention. 1 Introduction Commonsense is unstated background knowledge that is used to perceive, infer, and understand the physical world, human emotions, reactions, and knowledge of the common facts that most people agree with. Ordinary commonsense helps us to differentiate between simple false and true statements or answer questions, such as “can an elephant fit into the fridge” quickly, but they can be difficult for automatic systems (Davis, 2017). Recent advances in machine learning emphasize the importance of commonsense reasoning in natural language processing (NLP) and as a critical component of artificial intelligence (AI). In the fifty-year history of AI research, progress was extremely slow (Davis and Morgenstern, 2004) in automated commonsense reasoning. However, when transfer learning (Yosinski et al., 2014; Goodfellow et al., 2016) and then transformers were introduced to the NLP world (Vaswani et al., 2017), great breakthroughs and developments have occurred at an unprecedented pace (Pan and Yang, 2009; Tan et al., 2018). Advances in machine learning and deep learning methods have been achieved in numerous studies and wide range of disciplines (Panahi et al., 2019; Nemati et al., 2020; Sahour et al., 2020; Alshehri et al., 2020; Arodz and Saeedi, 2019; Oh et al., 2018). This paper describes a system participating in the SemEval-2020 “Commonsense Validation and Explanation (ComVE) Challenge”, multiple tasks of …

Content truncated to 500 words