On Commonsense Cues in BERT for Solving Commonsense Tasks

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 683–693 August 1–6, 2021. ©2021 Association for Computational Linguistics 683 On Commonsense Cues in BERT for Solving Commonsense Tasks Leyang Cui△†, Sijie Cheng‡, Yu Wu3, Yue Zhang†∗ △ Zhejiang University †School of Engineering, Westlake University ‡Fudan University 3Microsoft Research Asia cuileyang@westlake.edu.cn sjcheng20@fudan.edu.cn Wu.Yu@microsoft.com yue.zhang@wias.org.cn Abstract BERT has been used for solving common- sense tasks such as CommonsenseQA. While prior research has found that BERT does con- tain commonsense information to some extent, there has been work showing that pre-trained models can rely on spurious associations (e.g., data bias) rather than key cues in solving sen- timent classification and other problems. We quantitatively investigate the presence of struc- tural commonsense cues in BERT when solv- ing commonsense tasks, and the importance of such cues for the model prediction. Using two different measures, we find that BERT does use relevant knowledge for solving the task, and the presence of commonsense knowledge is positively correlated to the model accuracy. 1 Introduction Pre-trained language models (Peters et al., 2018; Radford et al., 2019; Devlin et al., 2019; Liu et al., 2019b) give competitive results on a variety of NLP tasks (Zhou and Zhao, 2019; Joshi et al., 2019; Liu and Lapata, 2019; Cui et al., 2020). It has been shown that they can effectively capture syntactic features (Goldberg, 2019), semantic information (Liu et al., 2019a) and factual knowledge (Petroni et al., 2019), which provides support for the success in downstream tasks. Recently, there has been some debate about whether commonsense knowledge can be learned by a language model trained on large corpora. While Davison et al. (2019), Bosselut et al. (2019) and Rajani et al. (2019) argue that pre-trained lan- guage models can directly identify commonsense facts, Lin et al. (2019) and Klein and Nabi (2019) believe that structured commonsense knowledge is not captured well. Pre-trained language models have achieved em- pirical success when fine-tuned on specific com- ∗Corresponding Author ������� �������������������� ���� ���������������� �� ����������������� �� �� ����� ������������������������� ���������������� �� �� ��� �� �� ������� ������������������������� ���������������� �� �� ����������� �� �� ������� ������������������������� ������������������� �� �� ������ �� �� ������� ������������������������� ����������������� �� �� ���������� �� �� �������������(����������� √ � � � � �������������(�����)(� ���� score Figure 1: Two methods used to study struc- tured commonsense knowledge in pre-trained Transformer. Commonsense link is drawn from the Target Concept (Answer Concept) to the Source Concept (Question Concept). monsense tasks such as COSMOS QA (Huang et al., 2019), SWAG (Zellers et al., 2018), and Common- senseQA (Talmor et al., 2019). One possible reason of the high performance is that there exist super- ficial cues or spurious associations in the dataset, which enables models to answer questions with- out understanding the task (Niven and Kao, 2019; Yu et al., 2020; Kaushik et al., 2020). For exam- ple, a model can choose the spurious cue word “meadow” as a feature for positive reviews sim- ply because “meadow” occurs frequently in posi- tive documents. It remains …

Content truncated to 500 words