Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2318–2327 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2318 Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks Yao Qiu, Jinchao Zhang ∗, Jie Zhou Pattern Recognition Center, WeChat AI, Tencent Inc, China {yasinqiu, dayerzhang, withtomzhou}@tencent.com Abstract Loading models pre-trained on the large-scale corpus in the general domain and fine-tuning them on specific downstream tasks is gradually becoming a paradigm in Natural Language Processing. Previous investigations prove that introducing a further pre-training phase be- tween pre-training and fine-tuning phases to adapt the model on the domain-specific unla- beled data can bring positive effects. How- ever, most of these further pre-training works just keep running the conventional pre-training task, e.g., masked language model, which can be regarded as the domain adaptation to bridge the data distribution gap. After observing di- verse downstream tasks, we suggest that differ- ent tasks may also need a further pre-training phase with appropriate training tasks to bridge the task formulation gap. To investigate this, we carry out a study for improving multi- ple task-oriented dialogue downstream tasks through designing various tasks at the further pre-training phase. The experiment shows that different downstream tasks prefer different fur- ther pre-training tasks, which have intrinsic correlation and most further pre-training tasks significantly improve certain target tasks rather than all. Our investigation indicates that it is of great importance and effectiveness to design appropriate further pre-training tasks model- ing specific information that benefit down- stream tasks. Besides, we present multiple constructive empirical conclusions for enhanc- ing task-oriented dialogues. 1 Introduction Pre-trained models, e.g., BERT (Devlin et al., 2019a), RoBERTa (Liu et al., 2019) and GPT2 (Radford et al., 2019), have been widely used in many NLP tasks. These models are pre- trained on the large-scale general text corpus, such as Wikipedia or books, with self-supervised train- ∗Jinchao Zhang is the corresponding author. ing objectives. Fine-tuning these models on down- stream tasks can achieve excellent performance. Recently, Gururangan et al. (2020) proposed a domain-adaptive pre-training method, they further pre-training the RoBERTa on a large corpus of unla- beled domain-specific text, e.g., biomedical papers and computer science papers, before fine-tuning on downstream tasks and achieved strong performance. Besides, they proved that it is also helpful to con- tinue pre-training on the task-specific text. Wu et al. (2020) applied this method to task-oriented dialogue and proposed a new self-supervised pre- training objective on dialogue corpus. Despite they achieved performance improvements, the improve- ments on different downstream tasks vary a lot, some tasks even obtain no improvement, which in- dicates that different downstream tasks may need different further pre-training tasks. To investigate this issue, we carry out experi- ments in the area of task-oriented dialogue. We choose one popular pre-training language model, BERT (Devlin et al., 2019a) as our base model, and construct a large scale domain-specific dialogue corpus which consists of nine task-oriented datasets for further pre-training (Wu et al., 2020). We also select …

Content truncated to 500 words