UAICS at SemEval-2020 Task 4: Using a Bidirectional Transformer for Task a

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 609–613 Barcelona, Spain (Online), December 12, 2020. 609 UAICS at SemEval-2020 Task 4: Using a bidirectional transformer for Task A Ciprian-Gabriel Cusmuliuc Lucia-Georgiana Coca ”Alexandru Ioan Cuza” University of Iasi, Faculty of Computer Science cusmuliuc.ciprian.gabriel@info.uaic.ro coca.lucia.georgiana@info.uaic.ro adiftene@info.uaic.ro Adrian Iftene Abstract Commonsense Validation and Explanation has been a difficult task for machines since the dawn of computing. Although very trivial to humans it poses a high complexity for machines due to the necessity of inference over a pre-existing knowledge base. In order to try and solve this problem the SemEval 2020 Task 4 - ”Commonsense Validation and Explanation (ComVE)” aims to evaluate systems capable of multiple stages of ComVE. The challenge includes 3 tasks (A, B and C), each with it’s own requirements. Our team participated only in task A which required selecting the statement that made the least sense. We choose to use a bidirectional transformer in order to solve the challenge, this paper presents the details of our method, runs and result. 1 Introduction Natural Language Understanding (NLU) is a topic that in recent years caught researchers’ attention. Growing interest has led to the development of multiple areas of study, one such area being represented by ”Commonsense Validation” which aims to evaluate a system’s ability to infer commonsense knowledge from basic sentences. The ”SemEval 2020 Task 4 - Commonsense Validation and Explanation (ComVE)” (Wang et al., 2020) seeks to research in the aforementioned field by testing systems that are able to differentiate between statements that make sense from the ones that don’t. The task is split in multiple tasks: task A implies selecting which statement is against commonsense between two very similar phrases, task B requires selecting between three reasons that explain why the statement from task A is incorrect and finally task C requires the machine to generate reasons why the statement from task A is against common sense. Our team participated only in task A, in the official results our system ranked 17th out of 45 submission with an accuracy of 89.1%, the best result being 97% and the worst 0%. The strategy was to use a pre-trained model, BERT (Devlin et al., 2018), in order to find out which sentence makes the most sense. This model has been trained and tested using different parameters in order to achieve the best result and it will be described with more details in section 2, ”System Overview”. Similar work has been done by (Trinh and Le, 2018) where they trained a RNN on a very large text collection resulting in a 63.7% accuracy in the Winograd Schema Challenge (Levesque et al., 2012) while (Kocijan et al., 2019) using a BERT model was able to achieve overall accuracy of 72.5%; this performance increase led us to choose the latter model as we feel it best fits our purpose. In this paper we will describe our system, results and conclusions. The structure is as follows: Section 2 will present the system …

Content truncated to 500 words