Team Solomon at SemEval-2020 Task 4: Be Reasonable: Exploiting Large-scale Language Models for Commonsense Reasoning

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 585–593 Barcelona, Spain (Online), December 12, 2020. 585 Team Solomon at SemEval-2020 Task 4: Be Reasonable: Exploiting large-scale language models for commonsense reasoning Vertika Srivastava∗ Rohit R.R Sudeep Kumar Sahoo∗ Mayank Raj Yeon Hyang Kim Ajay Jaiswal Samsung R&D Institute India, Bangalore {v.srivastava, sudeep.sahoo, purine.kim, rohit.r.r, mayank.raj, ajay.jaiswal}@samsung.com Abstract In this paper, we present our submission for SemEval 2020 Task 4 - Commonsense Validation and Explanation (ComVE). The objective of this task was to develop a system that can differentiate statements that make sense from the ones that don’t. ComVE comprises of three subtasks to challenge and test a system’s capability in understanding commonsense knowledge from various dimensions. Commonsense reasoning is a challenging task in the domain of natural language understanding and systems augmented with it can improve performance in various other tasks such as reading comprehension, and inferencing. We have developed a system that leverages commonsense knowledge from pretrained language models trained on huge corpus such as RoBERTa, GPT2, etc. Our proposed system validates the reasonability of a given statement against the backdrop of commonsense knowledge acquired by these models and generates a logical reason to support its decision. Our system ranked 2nd in subtask C with a BLEU score of 19.3, which by far is the most challenging subtask as it required systems to generate the rationale behind the choice of an unreasonable statement. In subtask A and B, we achieved 96% and 94% accuracy respectively standing at 4th position in both the subtasks. 1 Introduction In today’s digital age, information is shared widely in textual format via e-mails, news articles, social media posts and messages, blogs, internet forums, etc. We are now surrounded by textual information more than ever, which demands a meaningful understanding of a text by machines. Machines augmented with commonsense reasoning will be a key step towards achieving this. ‘Commonsense Knowledge’ also referred to as the background knowledge, is the understanding of the everyday world and the art of drawing inferences by manipulating the knowledge gathered. Humans are rational and have acquired a sense of reasoning by combining facts and beliefs from their day to day life. To an average person reasoning the fact that a person can have a pet dog but not pet dinosaur comes naturally and is fairly straightforward. Commonsense knowledge is assumed to be known to all and people typically tend to omit this while communicating with others, which makes it more challenging. Current Natural Language Understanding (NLU) systems assisted with semantic representations, statistical methods, and distributional representations have shown better performance than humans on many benchmarks but there is a growing concern that these systems scratch only the surface of the human level of understanding of the world and thus are too shallow. Natural language is complex in nature and NLU systems have tried to derive useful meaning by capturing the context i.e. neighboring words and sentences but these systems fail miserably when the context is restricted or omitted. …

Content truncated to 500 words