CN-HIT-IT.NLP at SemEval-2020 Task 4: Enhanced Language Representation with Multiple Knowledge Triples

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 494–500 Barcelona, Spain (Online), December 12, 2020. 494 CN-HIT-IT.NLP at SemEval-2020 Task 4: Enhanced Language Representation with Multiple Knowledge Triples Yice Zhang, Jiaxuan Lin, Yang Fan, Peng Jin, Yuanchao Liu, Bingquan Liu Intelligence Technology and Natural Language Processing Lab School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China {yczhang, linjiaxuan, yfan, pjin}@insun.hit.edu.cn {ycliu, liubq}@hit.edu.cn Abstract This paper describes our system that participated in the SemEval-2020 task 4: Commonsense Validation and Explanation. For this task, it is obvious that external knowledge, such as knowl- edge graph, can help the model understand commonsense in natural language statements. But how to select the right triples for statements remains unsolved, so how to reduce the interfer- ence of irrelevant triples on model performance is a research focus. This paper adopt a modi- fied K-BERT as the language encoder, to enhance language representation through triples from knowledge graphs. Experiments show that our method is better than models without external knowledge, and is slightly better than the original K-BERT. We got an accuracy score of 0.97 in subtaskA, ranking 1/45, and got an accuracy score of 0.948, ranking 2/35. 1 Introduction In recent years, language models trained on large scale corpora (Peters et al., 2018; Devlin et al., 2019; Lan et al., 2019) have performed exceptionally well on many benchmarks (Devlin et al., 2019), reaching or even surpassing human performance. It seems to show that Natural Language Understanding (NLU) is becoming easier. And the level of NLU may be seen in the ability to understand commonsense in natural language statements. Therefore, it is important to be able to evaluate how well a model can do for sense making(Wang et al., 2019). For more direct research on commonsense in natural language statements, Commonsense Validation and Explanation (ComVE) is proposed by Wang et al.(2020). The ComVE task consists of three subtasks, and we participated in subtaskA and subtaskB. SubtaskA is validation, requiring the model to identify which statement makes sense from the given two. Then for the against-common-sense statement, three optional sentences are provided to explain why the statement does not make sense. In subtaskB, named Explanation (multi-choice), the only one correct reason is required to be identified from two other con- fusing ones. Intuitively, knowledge graphs (KGs) can help language models understand commonsense. For exam- ple, for the sentence, “all whales are small”, the triple <whale, hasproperty, big> are useful. Under such motivation, K-BERT (Liu et al., 2019) injects triples into sentences as domain knowledge. However, how to select the helpful triples from KG remains a problem. When we fuse too much external knowledge, irrelevant knowledge will adversely affect the model, which is called knowledge noise (KN) issue. To overcome KN issue, K-BERT introduces soft-position and visible matrix to limit the influence of the triples. But KN issue is still serious, especially when the number of injected triples increases. Therefore, this paper introduce a variant of K-BERT for further reduction of KN. Besides, we …

Content truncated to 500 words