Retrieval Enhanced Model for Commonsense Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 303–313 August 1–6, 2021. ©2021 Association for Computational Linguistics 303 Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning Benjamin Minixhofer1∗, Milan Gritta2 and Ignacio Iacobacci2 1Johannes Kepler University, Linz, Austria 2Huawei Noah’s Ark Lab, London, UK bminixhofer@gmail.com {milan.gritta, ignacio.iacobacci}@huawei.com Abstract Transfer learning has become the dominant paradigm for many natural language process- ing tasks. In addition to models being pre- trained on large datasets, they can be further trained on intermediate (supervised) tasks that are similar to the target task. For small Nat- ural Language Inference (NLI) datasets, lan- guage modelling is typically followed by pre- training on a large (labelled) NLI dataset be- fore fine-tuning with each NLI subtask. In this work, we explore Gradient Boosted De- cision Trees (GBDTs) as an alternative to the commonly used Multi-Layer Perceptron (MLP) classification head. GBDTs have de- sirable properties such as good performance on dense, numerical features and are effective where the ratio of the number of samples w.r.t the number of features is low. We then intro- duce FreeGBDT, a method of fitting a GBDT head on the features computed during fine- tuning to increase performance without addi- tional computation by the neural network. We demonstrate the effectiveness of our method on several NLI datasets using a strong baseline model (RoBERTa-large with MNLI pretrain- ing). The FreeGBDT shows a consistent im- provement over the MLP classification head. 1 Introduction Recent breakthroughs in transfer learning ranging from semi-supervised sequence learning (Dai and Le, 2015) to ULMFiT (Howard and Ruder, 2018), ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) have brought significant improvements to many natural language processing (NLP) tasks. Transfer learning involves pretraining neural networks, often based on the Transformer (Vaswani et al., 2017), on large amounts of text in a self-supervised manner in order to learn ∗Work conducted as Research Intern at Huawei Noah’s Ark Lab. transferable language features useful for many NLP tasks. Pretraining is followed by fine-tuning the model on the target task. Pretrained models can also be further trained on intermediate labelled datasets which are similar to the target task before the final fine-tuning stage (Pruksachatkun et al., 2020). We refer to this as intermediate supervised pretraining. In this manner, the network learns more meaningful internal representations of the input text that are better aligned with the target task. In order to fine-tune the pretrained network, some latent representation of the text (e.g. the hidden state corresponding to the [CLS] token of BERT-like models) is used as the input to a classification head, usually a randomly initialised Multi-Layer Perceptron (MLP) (Wolf et al., 2019). The output of the classification head can then be interpreted as probability distribution over classes. The input to the classification head is referred to as features throughout the paper. It is a high-dimensional vector that serves as a rich, distributed representation of the input text. We investigate whether replacing the commonly used MLP classification head with a GBDT …

Content truncated to 500 words