A Systematic Investigation of Commonsense Knowledge in Large Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11838 - 11855 December 7-11, 2022 ©2022 Association for Computational Linguistics A Systematic Investigation of Commonsense Knowledge in Large Language Models Xiang Lorraine Li† ∗ Adhiguna Kuncoro‡ Jordan Hoffmann⋆♢ Cyprien de Masson d’Autume♦♢ Phil Blunsom▲♣♢ Aida Nematzadeh‡ † Allen Institute for Artificial Intelligence ‡ DeepMind ⋆ Inflection AI ♦ Reka ▲ Cohere ♠ University of Oxford lorrainel@allenai.org nematzadeh@google.com Abstract Language models (LMs) trained on large amounts of data (e.g., Brown et al., 2020; Pat- wary et al., 2021) have shown impressive per- formance on many NLP tasks under the zero- shot and few-shot setup. Here we aim to better understand the extent to which such models learn commonsense knowledge — a critical component of many NLP applications. We con- duct a systematic and rigorous zero-shot and few-shot commonsense evaluation of large pre- trained LMs, where we: (i) carefully control for the LMs’ ability to exploit potential surface cues and annotation artefacts, and (ii) account for variations in performance that arise from factors that are not related to commonsense knowledge. Our findings highlight the limi- tations of pre-trained LMs in acquiring com- monsense knowledge without task-specific su- pervision; furthermore, using larger models or few-shot evaluation are insufficient to achieve human-level commonsense performance. 1 Introduction Common sense — the implicit knowledge about everyday situation that is shared by humans — is an important prerequisite for developing general- purpose intelligent systems (McCarthy et al., 1960; Liu and Singh, 2004; Gunning, 2018). Intrigu- ingly, recent large language models (LMs, Brown et al., 2020; Patwary et al., 2021; Rae et al., 2021) have achieved remarkable performance at various common sense benchmarks (e.g., Sakaguchi et al., 2020; Zellers et al., 2019a; Bisk et al., 2020b; Sap et al., 2019b), even when they are evaluated in a zero-shot or few-shot fashion, without explicit commonsense supervision. We revisit this apparent success, and conduct a rigorous study to better un- derstand the extent to which such pre-trained LMs are able to capture commonsense knowledge. ∗ Work done during DeepMind internship when Lorraine was a PhD student at UMass Amherst. ♢ Work done at Deep- Mind Figure 1: The experiment settings with their correspond- ing input to the LM. The example is taken from Social IQa (Sap et al., 2019b) where we convert questions to natural text using the rules of Shwartz et al. (2020); this conversion yields to better performance (§5). In this work, we focus on zero- and few- shot evaluations of pre-trained LMs without commonsense-specific fine-tuning for two reasons: First, we aim to examine if a pre-trained LM is able to acquire general commonsense knowledge. As pre-trained LMs constitute a foundational build- ing block of NLP today, any deficiencies in their commonsense understanding can thus adversely manifest in downstream applications (Bommasani et al., 2021). Fine-tuning the LM would make it hard to disentangle how much of the commonsense knowledge is acquired by the underlying LM, as op- posed to the task-specific supervision from a …

Content truncated to 500 words