Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1121–1133 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1121 Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models Tianxing He MIT tianxing@mit.edu Jun Liu Facebook junliu@fb.com Kyunghyun Cho New York University kyunghyun.cho@nyu.edu Myle Ott & Bing Liu Facebook {myleott,bingl}@fb.com James Glass MIT glass@mit.edu Fuchun Peng Facebook fuchunpeng@fb.com Abstract In this work, we study how the finetuning stage in the pretrain-finetune framework changes the behavior of a pretrained neural language gen- erator. We focus on the transformer encoder- decoder model for the open-domain dialogue response generation task. Our major finding is that after standard finetuning, the model forgets some of the important language gen- eration skills acquired during large-scale pre- training. We demonstrate the forgetting phe- nomenon through a set of detailed behavior analysis from the perspectives of knowledge transfer, context sensitivity, and function space projection. As a preliminary attempt to allevi- ate the forgetting problem, we propose an intu- itive finetuning strategy named “mix-review”. We find that mix-review effectively regularizes the finetuning process, and the forgetting prob- lem is alleviated to some extent. Finally, we discuss interesting behavior of the resulting di- alogue model and its implications. 1 Introduction Large-scale unsupervised pretraining (Peters et al., 2018; Devlin et al., 2018; Song et al., 2019; Yang et al., 2019; Liu et al., 2019) has recently been shown to greatly boost the performance of nat- ural language processing (NLP) models. On a high level, the pretrain-finetune framework can be viewed as a simple two-stage procedure: (1) Use large-scale unsupervised text data to pretrain the model; (2) Use target task data to finetune the model. Recently, multiple works (Radford et al., 2019; Jiang et al., 2020; Roberts et al., 2020; Talmor et al., 2019) have reported that pretrained language mod- els (LM) have implicitly stored large amounts of “world knowledge” in its parameters, and are able to answer common-sense questions. While these studies are encouraging, during the finetuning stage Figure 1: During finetuning, the model’s performance on the pretraining data drastically degrades. the model is usually trained on a dataset that is very different from the pretraining data, which leads to the potential danger that the model could forget precious skills gained during pretraining. This is an important question for open-domain dialogue re- sponse generation, which is the focus of our work, because the knowledge acquired during pretrain- ing can greatly help make the dialogue interaction more engaging or informative. In Figure 1, we show that during finetuning, the model’s performance on the pretraining data dras- tically degrades. While this drop is concerning, it does not necessarily mean that the skills from pretrained model are not well transferred to the end dialogue task, because the model should be evaluated in a dialogue setting. To better answer the question about how fine- tuning changes the pretrained model’s behavior, in this work we conduct a set of behavior analysis from the perspectives of knowledge …

Content truncated to 500 words