JBNU at SemEval-2020 Task 4: BERT and UniLM for Commonsense Validation and Explanation

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 527–534 Barcelona, Spain (Online), December 12, 2020. 527 JBNU at SemEval-2020 Task 4: BERT and UniLM for Commonsense Validation and Explanation Jong-Hyeon Lee and Seung-Hoon Na Computer Science and Engineering, Jeonbuk National University, South Korea {gus8423,nash}@jbnu.ac.kr Abstract This paper presents our contributions to the SemEval-2020 Task 4 Commonsense Validation and Explanation (ComVE) and includes the experimental results of the two Subtasks B and C of the SemEval-2020 Task 4. Our systems rely on pre-trained language models, i.e., BERT (including its variants) and UniLM, and rank 10th and 7th among 27 and 17 systems on Subtasks B and C, respectively. We analyze the commonsense ability of the existing pretrained language models by testing them on the SemEval-2020 Task 4 ComVE dataset, specifically for Subtasks B and C, the explanation subtasks with multi-choice and sentence generation, respectively. 1 Introduction SemEval-2020 Task 4 aims to evaluate whether a system identifies and rationalizes a given natural language statement to be comprehensible under commonsense knowledge (Wang et al., 2020). Starting from the pilot study (Wang et al., 2019), SemEval-2020 Task 4 consists of three subtasks: 1) Subtask A: differentiating statements that make sense from those that do not, 2) Subtask B and C: selecting a reason or explaining why the statement does not make sense. This paper presents an overview of our systems that were examined for Subtasks B and C, as well as the final results. Recently, inspired from the success of ELMo (Peters et al., 2018), GPT (Radford et al., 2018), and BERT (Devlin et al., 2019) which demonstrated considerable improvements on various language understanding tasks, including the GLUE benchmark (Wang et al., 2018) and SQuAD (Rajpurkar et al., 2016), significant studies have been conducted on pretrained language models, including RoBERTa and XLNet, which efficiently reduce the training and inference costs (Liu et al., 2019; Lan et al., 2019), Transformer-XL, which extends to large contexts (Dai et al., 2019), XLNet (Yang et al., 2019), MASS and UniLM, which support pretrained language models for generation tasks (Song et al., 2019; Dong et al., 2019), DistillBERT for lightweight pretrained models (Sanh et al., 2019), analyzing BERT’s syntactic knowledge (Hewitt and Manning, 2019), knowledge-enhanced language models (Peters et al., 2019), cross-lingual pretrained language models (Conneau and Lample, 2019), few-shot learning using language models (Radford et al., 2019; Brown et al., 2020), scaling up pretrained language models (Raffel et al., 2019) and the retrieval-based language model using dense retrieval on the external corpus (Guu et al., 2020). A critical review of the pretrained language models that include classical neural language models (Bengio et al., 2003) has been presented in (Young et al., 2018). Given the era of pretrained language models, the main design goal of our system on ComVE is to explore the effect of using pretrained language models on Subtasks B and C, both for the understanding and generation tasks. In the top-level design, BERT is selected for Subtask B, and UniLM, a generalized pretrained language …

Content truncated to 500 words