KaLM at SemEval-2020 Task 4: Knowledge-aware Language Models for Comprehension and Generation

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 543–550 Barcelona, Spain (Online), December 12, 2020. 543 KaLM at SemEval-2020 Task 4: Knowledge-aware Language Models for Comprehension And Generation Jiajing Wan* Xinting Huang* Sun Yat-Sen University Guangzhou, China {wanjj, huangxt39}@mail2.sysu.edu.cn Abstract This paper presents our strategies in SemEval 2020 Task 4: Commonsense Validation and Explanation. We propose a novel way to search for evidence and choose the different large-scale pre-trained models as the backbone for three subtasks. The results show that our evidence- searching approach improves model performance on commonsense explanation task. Our team ranks 2nd in subtask C according to human evaluation score.1 1 Introduction Commonsense reasoning has been seen as one of the key ability for intelligent machines to perform various activities (Davis and Marcus, 2015). SemEval 2020 Task 4 (Wang et al., 2020) is a commonsense validation and explanation task which is inspired by Wang et al. (2019). This task consists of three subtasks. The first subtask is to choose the one which makes sense from two natural language statements with similar wordings; The second subtask is to decide among three options the most crucial reason why a given statement does not make sense; The third subtask requires the machine to generate the reasons. To make predictions or generate reasons, background knowledge is essential. A simple way to supplement that knowledge is utilizing plain texts from natural language databases, e.g. Wikipedia. Intuitively, for a specific given statement, plain texts can be provided by searching similar sentences in the database. In other words, each evidence sentences contain the keywords of the given statement. This method is used in some of the state-of-the-art models (Lv et al., 2019). However, for the purpose of explaining the reason, evidence which has similar wording with the given statement may lack information, and sometimes can be misleading when the given statement does not make sense. In this work, we propose a novel way for evidence-searching using plain texts. We obtain evidence by searching for the meaning of the keywords in the given statement. In other words, using evidence containing the meanings of the keywords rather than containing the keywords themselves. The reason for our method is that these meanings may provide important information to explain why a statement makes sense or does not make sense. For example in Figure 1, the definition of ”aircraft carrier”—”A warship designed to carry aircraft”—is given by the evidence obtained from the database. Such a definition well explains why the given statement is wrong and can be used when generating the reason. In contrast, evidence containing both ”aircraft carrier” and ”human” will not be helpful. We conduct experiments on subtask A, B, and C. Results show that our evidence-searching method boosts the performance on subtask C. Our team achieves accuracy of 95.3 (9th place) in subtask A and 93.2 (7th place) in subtask B. In subtask C, Our approach achieves the BLEU score of 18.5 (3rd place) and human evaluation score of 2.08 (2nd place). Moreover, the …

Content truncated to 500 words