Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5474–5483 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5474 Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models Kaixin Ma†, Filip Ilievski§, Jonathan Francis†¶, Satoru Ozaki†, Eric Nyberg†, Alessandro Oltramari¶ †Language Technologies Institute, Carnegie Mellon University §Information Sciences Institute, University of Southern California ¶Human-Machine Collaboration, Bosch Research Pittsburgh {kaixinm, jmf1, sozaki, ehn}@cs.cmu.edu, ilievski@isi.edu, alessandro.oltramari@us.bosch.com Abstract Commonsense reasoning benchmarks have been largely solved by fine-tuning language models. The downside is that fine-tuning may cause models to overfit to task-specific data and thereby forget their knowledge gained dur- ing pre-training. Recent works only propose lightweight model updates as models may al- ready possess useful knowledge from past ex- perience, but a challenge remains in under- standing what parts and to what extent mod- els should be refined for a given task. In this paper, we investigate what models learn from commonsense reasoning datasets. We mea- sure the impact of three different adaptation methods on the generalization and accuracy of models. Our experiments with two models show that fine-tuning performs best, by learn- ing both the content and the structure of the task, but suffers from overfitting and limited generalization to novel answers. We observe that alternative adaptation methods like prefix- tuning have comparable accuracy, but general- ize better to unseen answers and are more ro- bust to adversarial splits. 1 Introduction Machine commonsense reasoning has recently gained new traction, largely due to a collection of diverse benchmarks (Talmor et al., 2019; Bhaga- vatula et al., 2019; Sap et al., 2019) and the success- ful application of language modeling methods on these benchmarks (Ma et al., 2019; Shwartz et al., 2020; Bauer and Bansal, 2021). The most widely adopted approach to solve these commonsense rea- soning tasks is by fine-tuning large pre-trained lan- guage models (LMs) (Devlin et al., 2019; Liu et al., 2019) on the task-specific training data. Mean- while, it has been shown that language models are able to acquire certain commonsense background knowledge, during their pre-training on large tex- tual data (Petroni et al., 2019; Davison et al., 2019; Ma et al., 2021). In light of these findings and the large capacity of these language models, recent work has proposed lightweight alternatives to fine- tuning LMs, e.g., by only updating a small amount of additional parameters (Lin et al., 2020b; Li and Liang, 2021), or by updating the inputs while keep- ing the model weights intact (Jiang et al., 2020; Shin et al., 2020). Intuitively, these lightweight methods may retain the model’s pre-trained knowl- edge to a large extent, and elicit the suitable knowl- edge for the target task, provided that much of this knowledge has already been encoded in the model parameters. However, to our knowledge, no com- prehensive comparison exists between these model updating strategies. In this paper, we pose the question: What do mod- els learn from commonsense reasoning datasets? We consider three representative learning methods: regular fine-tuning, model extension with prefix- tuning …

Content truncated to 500 words