BUT-FIT at SemEval-2020 Task 4: Multilingual Commonsense

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 374–390 Barcelona, Spain (Online), December 12, 2020. 374 BUT-FIT at SemEval-2020 Task 4: Multilingual commonsense Josef Jon, Martin Fajcik, Martin Docekal, Pavel Smrz Brno University of Technology, Faculty of Information Technology 612 66 Brno, Czech Republic {ifajcik,ijon,idocekal,smrz}@fit.vutbr.cz Abstract This paper describes work of the BUT-FIT’s team at SemEval 2020 Task 4 - Commonsense Validation and Explanation. We participated in all three subtasks. In subtasks A and B, our submissions are based on pretrained language representation models (namely ALBERT) and data augmentation. We experimented with solving the task for another language, Czech, by means of multilingual models and machine translated dataset, or translated model inputs. We show that with a strong machine translation system, our system can be used in another language with a small accuracy loss. In subtask C, our submission, which is based on pretrained sequence-to-sequence model (BART), ranked 1st in BLEU score ranking, however, we show that the correlation between BLEU and human evaluation, in which our submission ended up 4th, is low. We analyse the metrics used in the evaluation and we propose an additional score based on model from subtask B, which correlates well with our manual ranking, as well as reranking method based on the same principle. We performed an error and dataset analysis for all subtasks and we present our findings. 1 Introduction Commonsense knowledge is collection of facts, that an average human being is expected to know and be able to reason with these facts. It includes the ability to asses physical qualities, behavior and purpose of inanimate objects, animals and people. In other words, it is an implicit knowledge, something we feel we do not have to explain. Since this topic is among the most important challenges in artificial intelligence, there is a large number of publications regarding commonsense – see for example books by Mueller (2014) or Davis (2014). The goal of SemEval 2020 Task 4 - Commonsense Validation and Explanation (Wang et al., 2020) was to asses how current NLP approaches compare with humans in terms of such knowledge. Three subtasks were devised to determine the ability of the participating systems to reason within commonsense knowledge in English. In the first subtask, the goal was to differentiate between statements that make sense and those that do not. In the second and third subtask, only nonsensical statements were considered, and the assignment was to explain why are the statements against common sense. In subtask number two, the systems were presented with three possible choices for the explanation, while in the third subtask, the systems were expected to generate the explanation from scratch. A more detailed description of the tasks is provided in the task paper. Our submissions are based on recent pretrained models, namely ALBERT (Lan et al., 2019) and BART (Lewis et al., 2019). We experimented with data augmentation using round trip translations of the datasets and also with machine translation of either the queries, or the whole dataset into …

Content truncated to 500 words