Training Dynamics for Text Summarization Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 2061 - 2073 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Training Dynamics for Text Summarization Models Tanya Goyal1 Jiacheng Xu1 Junyi Jessy Li2 Greg Durrett1 1 Department of Computer Science 2 Department of Linguistics The University of Texas at Austin tanyagoyal@utexas.edu Abstract Pre-trained language models (e.g. BART) have shown impressive results when fine-tuned on large summarization datasets. However, lit- tle is understood about this fine-tuning pro- cess, including what knowledge is retained from pre-training time or how content selec- tion and generation strategies are learnt across iterations. In this work, we analyze the train- ing dynamics for generation models, focusing on summarization. Across different datasets (CNNDM, XSUM, MEDIASUM) and summary properties, such as abstractiveness and hallu- cination, we study what the model learns at different stages of its fine-tuning process. We find that a propensity to copy the input is learned early in the training process consis- tently across all datasets studied. On the other hand, factual errors, such as hallucination of unsupported facts, are learnt in the later stages, though this behavior is more varied across do- mains. Based on these observations, we ex- plore complementary approaches for modify- ing training: first, disregarding high-loss to- kens that are challenging to learn and second, disregarding low-loss tokens that are learnt very quickly in the latter stages of the training process. We show that these simple training modifications allow us to configure our model to achieve different goals, such as improving factuality or improving abstractiveness.1 1 Introduction Transformer-based pre-training (Lewis et al., 2020; Zhang et al., 2020) has led to substantial improve- ments in the performance of abstractive summa- rization models. This pre-training and fine-tuning paradigm has been widely studied with respect to what training datasets, model sizes and other hy- perparameters are needed to optimize task-specific evaluation metrics, such as perplexity or ROUGE 1Code and all model checkpoints are available at https://github.com/tagoyal/ training-dynamics-generation. for text generation. However, abstractive summa- rization is a complex task involving several com- ponents such as content selection and rewriting that are performed implicitly by end-to-end models such as BART (Lewis et al., 2020) or PEGASUS (Zhang et al., 2020). Currently, we have little in- sight into this aspect of the fine-tuning process, namely what “skill” or behavior is learnt at which stage of the training process. Recent work (Schuster et al., 2019; Utama et al., 2020a) has studied training dynamics for sequence classification tasks such as NLI and fact verifica- tion, demonstrating how these can be leveraged to mitigate dataset biases. However, text generation is a substantially different task from classification, due to the sequential nature of predictions and the mismatch between teacher-forced training and in- ference time. The nature of the training process and potential interventions to modify what gets learned are poorly understood. In this paper, we make the first attempt at under- standing the fine-tuning process of large pre-trained language models for summarization. We study two essential components of abstractive summarization …

Content truncated to 500 words