KDE SenseForce at SemEval-2020 Task 4: Exploiting BERT for Commonsense Validation and Explanation

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 551–555 Barcelona, Spain (Online), December 12, 2020. 551 KDE SenseForce at SemEval-2020 Task 4: Exploiting BERT for Commonsense Validation and Explanation Khanddorj Mendbayar, Masaki Aono Department of Computer Science Toyohashi University of Technology Toyohashi, Aichi, Japan khanddorj@kde.cs.tut.ac.jp, aono@tut.jp Abstract Using a natural language understanding system for commonsense comprehension is getting increasing attention from researchers. Current multi-purpose state-of-the-art models suffer on commonsense validation and explanation tasks. We have adopted one of the state-of-the-art models and proposing a method to boost the performance of the model in commonsense related tasks. 1 Introduction Commonsense knowledge is a knowledge that is built by everyday activities and vital for living. Because of the assumption that every person has the commonsense, we tend to discard the commonsense from social communications. Hence, most of the text data lack explicit declaration of commonsense (Liu and Singh, 2004). Nonetheless, basic knowledge of the commonsense related to space, physical interactions, and people is necessary for the development of the natural language processing, computer vision, and robotics (Davis and Marcus, 2015), and has been studied as early as 1956 (Mueller, 2014). Currently, there are many natural language processing subfields where machine learning has started to outperform humans (Wang et al., 2017). However, even the current state-of-the-art models like BERT (Devlin et al., 2018), Elmo (Peters et al., 2018) fail in commonsense validation and explanation tasks (Wang et al., 2019). We propose a system to cope with commonsense validation and explanation tasks using input transfor- mation and word feature scaling. Our proposed system extracts features about every word in the sentence from its surrounding contexts except the word itself then compares extracted features with the original word features to get a score map ( usually, scaling coefficient is called ‘attention ’, whereas it is called score map to avoid a confusion ). Then the score map is used to scale the word feature in the current state-of-the-art models. The result is noticeable, considering that the system uses only one additional fully connected layer on top of the existing system. For the Validation (A) and Explanation (B) task, the proposed system outperforms the result of the BERT by 9.5% and 27.2% , respectively. The results are submitted to the SemEval-2020 (Wang et al., 2020) and turned out to be in 33rd and 22nd places respectively. The system has room for improvement, since the tested input transformation works on only word level. 2 Background Nowadays, Transformer’s encoder and decoder model (Vaswani et al., 2017) has been used as the basis for almost all state-of-the-art technologies in natural language processing. A typical example is BERT. BERT is pretrained on a large dataset to predict masked words in a sentence from the context. This pretraining method creates a discrepancy between pretraining and finetuning (Yang et al., 2019). We think this pretraining method is used insufficiently in the commonsense validation and explanation tasks considering the pretraining dataset contains some commonsense already. To use this pretraining method better, …

Content truncated to 500 words