Does QA-based intermediate training help fine-tuning language models for text classification?

Does QA-based intermediate training help fine-tuning language models for text classification? Shiwei Zhang School of Computing Technologies RMIT University, Australia dr.shiwei.zhang@gmail.com Xiuzhen Zhang∗ School of Computing Technologies RMIT University, Australia xiuzhen.zhang@rmit.edu.au Abstract Fine-tuning pre-trained language models for downstream tasks has become a norm for NLP. Recently it is found that intermediate train- ing based on high-level inference tasks such as Question Answering (QA) can improve the performance of some language models for tar- get tasks. However it is not clear if intermedi- ate training generally benefits various language models. In this paper, using the SQuAD-2.0 QA task for intermediate training for target text classification tasks, we experimented on eight tasks for single-sequence classification and eight tasks for sequence-pair classification using two base and two compact language mod- els. Our experiments show that QA-based in- termediate training generates varying transfer performance across different language models, except for similar QA tasks. 1 Introduction The framework of fine-tuning pre-trained Language models (LMs), especially transformer-based LMs, for downstream tasks has shown state-of-the-art performance on many natural language process- ing (NLP) tasks (Devlin et al., 2019; Raffel et al., 2020). It is believed that the pre-training stage leads LMs to develop general-purpose abilities and knowledge that can then be transferred to down- stream tasks (Raffel et al., 2020). To further improve the performance of pre- trained LMs on target tasks, two novel training approaches have been recently researched, namely further pre-training and intermediate training. A further pre-training stage for LMs (Gururangan et al., 2020) is a stage between pre-training and fine-tuning, which further pre-trains LMs on an extra dataset using unsupervised objectives. It has been found that further pre-training LM on the tar- get domain (domain-adaptive pre-training) leads to ∗Corresponding author. Language Model SQuAD-2.0 Text Classification Pre-training Intermediate Training Fine-tuning Knowledge Transfer Knowledge Transfer Figure 1: We experiment SQuAD-2.0 as the intermedi- ate training task for text classification tasks. improved performance on target tasks (Gururangan et al., 2020). Another effective transfer learning approach named intermediate training that chooses to train a LM model on an intermediate task via supervised manner and then fine-tune it on target tasks. This also leads to promising results across various NLP tasks including text classification, QA and sequence labeling (Phang et al., 2018; Vu et al., 2020; Pruksachatkun et al., 2020). Text classification is the problem of classify- ing text into categories or classes which has been widely studied. In terms of input, there are mainly two forms of text classification problems: single- sequence classification tasks (e.g., sentiment classi- fication and topic classification) and pairwise tasks (e.g., NLI and IR-related QA). In recent years, a common approach to tackle text classification prob- lems is to fine-tune a pre-trained LM on target text classification tasks. Recently, advanced transfer learning-based approaches have been proposed to further improve the performance. For example, a re- cent work (Sun et al., 2019) has studied how to fine- tune BERT for text classification. They found that further pre-training LM using data within-task or in-domian can improve …

Content truncated to 500 words