Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3878–3887 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3878 Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning Alexandre Tamborrino∗, Nicola Pellican`o∗, Baptiste Pannier∗, Pascal Voitot, Louise Naudin Samsung Strategy and Innovation Center {a.tamborrino,n.pellicano,b.pannier,p.voitot,l.naudin}@samsung.com Abstract Fine-tuning of pre-trained transformer models has become the standard approach for solv- ing common NLP tasks (Devlin et al., 2019). Most of the existing approaches rely on a ran- domly initialized classifier on top of such net- works. We argue that this fine-tuning proce- dure is sub-optimal as the pre-trained model has no prior on the specific classifier labels, while it might have already learned an intrinsic textual representation of the task. In this paper, we introduce a new scoring method that casts a plausibility ranking task in a full-text format and leverages the masked language modeling head tuned during the pre-training phase. We study commonsense reasoning tasks where the model must rank a set of hypotheses given a premise, focusing on the COPA (Gordon et al., 2012), Swag (Zellers et al., 2018), HellaSwag (Zellers et al., 2019) and CommonsenseQA (Talmor et al., 2019) datasets. By exploiting our scoring method without fine-tuning, we are able to produce strong baselines (e.g. 80% test accuracy on COPA) that are comparable to supervised approaches. Moreover, when fine-tuning directly on the proposed scoring function, we show that our method provides a much more stable training phase across ran- dom restarts (e.g ×10 standard deviation re- duction on COPA test accuracy) and requires less annotated data than the standard classifier approach to reach equivalent performances. 1 Introduction Recent advances in natural language processing have been made using sequential transfer learning over large pre-trained transformer models. From these models, most NLP tasks can be addressed by adding a classifier on top of the transformer embedding outputs (Devlin et al., 2019; Liu et al., 2019) . ∗Equal contribution. In this paper, we tackle a subset of NLP tasks consisting in plausibility ranking. Such tasks can be formalised as follows: given a unique premise p and a set of hypotheses H = {hi}i=1...n, the task consists in returning the appropriate hypothe- sis h∗ ∈ H that matches p (see Section 3 for more details). A natural task that fits into this problem formulation is commonsense reasoning. Thus, it will be the main focus of the present paper. Traditionally, this problem is solved by jointly classifying each pair (p, hi)i=1...n. For instance, assuming a Masked Language Modeling (MLM) model is used, an example from the COPA dataset (Gordon et al., 2012) is commonly casted into two distinct examples: • [CLS] The man broke his toe. [SEP] He dropped a hammer on his foot. [SEP] → correct • [CLS] The man broke his toe. [SEP] He got a hole in his sock. [SEP] → incorrect The special token [CLS] (used for sentence level tasks) is then provided to a classifier in order to predict the label of …

Content truncated to 500 words