DEEPYANG at SemEval-2020 Task 4: Using the Hidden Layer State of BERT Model for Differentiating Common Sense

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 516–520 Barcelona, Spain (Online), December 12, 2020. 516 DEEPYANG at SemEval-2020 Task 4: Using the Hidden Layer State of BERT model for Differentiating Common Sense Yang Bai, Xiaobing Zhou* School of Information Science and Engineering Yunnan University, Yunnan, P.R. China Abstract Introducing common sense to natural language understanding systems has received increasing research attention. To facilitate the researches on common sense reasoning, the SemEval-2020 Task 4 Commonsense Validation and Explanation(ComVE) is proposed. We participate in sub-task A and try various methods including traditional machine learning methods, deep learning methods, and also recent pre-trained language models. Finally, we concatenate the original output of BERT and the output vector of BERT hidden layer state to obtain more abundant semantic information features, and obtain competitive results. Our model achieves an accuracy of 0.8510 in the final test data and ranks 25th among all the teams. 1 Introduction Humans can utilize a variety of knowledge and reasoning to help understand the meanings of language. For example, it is easy for a human to know that someone can put a turkey into a refrigerator. Still, he can never put an elephant into it with basic commonsense reasoning, but it can be non-trivial for a system to tell the difference(Wang et al., 2019). What enables us to arrive at this conclusion is the knowledge we have about the world and the underlying reasoning process, often called commonsense thought (Minsky, 2000) or commonsense reasoning(Davis and Marcus, 2015), that allows us to connect pieces of knowledge to reach the new conclusion. We know that the refrigerator is household appliances and turkeys are the food we usually eat, but elephants cannot be eaten; the elephant is larger than the refrigerator and cannot be put in it. While this kind of knowledge and reasoning becomes so naturally to humans, it is extremely difficult for machines. Despite significant advances in natural language processing in the last several decades, machines are still far away from having this type of natural language inference (NLI) ability(Storks et al., 2019). Therefore, we need an effective automatic method to differentiate natural language statements that make sense from those that do not make sense. In this paper, the used methods and the results obtained by our team, DEEPYANG, on the ComVE shared task organized at SemEval 2020 are presented(Wang et al., 2020). The ComVE shared task includes three sub-tasks Commonsense Validation, Commonsense Explanation (Multi-Choice), and Commonsense Explanation (Generation). We only participate in subtask A. The main goal in sub-task A is to choose which statement of the two is against common sense. By definition, the sub-task A is to select from two natural language statements with similar wordings which one makes sense and which one does not make sense. For this subtask, we try traditional machine learning methods, deep learning methods, and pre-trained models. After comparison, we choose the BERT-base model and concatenate the original output of BERT and the output vector of BERT hidden layer to …

Content truncated to 500 words