When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3206–3219 May 2-6, 2023 ©2023 Association for Computational Linguistics When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization Faisal Ladhak1, Esin Durmus2, Mirac Suzgun2, Tianyi Zhang2 Dan Jurafsky2, Kathleen McKeown1, Tatsunori Hashimoto2 1Columbia Univeristy 2Stanford Univeristy Abstract Large language models (LLMs) are subject to sociocultural and other biases previously identi- fied using intrinsic evaluations. However, when and how these intrinsic biases in pre-trained LM representations propagate to downstream, fine-tuned NLP tasks like summarization is not well understood. In this work, we investigate one type of bias—name-nationality bias—and trace it from the pre-training stage to a down- stream summarization task across multiple sum- marization modeling choices. We show that these biases manifest themselves as hallucina- tions in summarization, leading to factually in- correct summaries. We also find that this prop- agation of biases is algorithm-dependent: more abstractive models allow biases to propagate more directly to downstream tasks as halluci- nated facts. Building on these observations, we further analyze how changes to the adaptation method and fine-tuning data set affect name nationality biases and show that while they can reduce the overall rate of hallucinations, they do not change the types of biases that do appear. 1 Introduction Fine-tuning pre-trained large language models (LLMs) has recently become the de facto approach to building effective text summarization systems (Devlin et al., 2019; Zhang et al., 2019; Lewis et al., 2020). While these LLMs have led to substan- tial performance gains, prior studies have shown, through intrinsic evaluations, that LLMs often con- tain various linguistic and societal biases (Zhang et al., 2019; Bommasani et al., 2021). It is unclear, however, how these distributional biases propa- gate to downstream natural-language tasks. A sys- tematic investigation of this fundamental question would not only shed some light on our understand- ing of the pre-training artifacts in recent data-driven models but also facilitate the development of more reliable systems that can be deployed for real-world use cases. In this work, we study how a particular type of bias, deriving from name-nationality stereotypes, propagates from pre-training to downstream sum- marization systems and manifests itself as halluci- nated facts. Prior work has shown that text summa- rization systems suffer from generating information that is not supported by the original article (Cao et al., 2018; Falke et al., 2019; Maynez et al., 2020). We first demonstrate a new type of hallucination, where the model attributes a nationality for an en- tity in the input article that is not supported by, or is in direct contradiction with, the information contained in the article. We then present a new out- of-distribution evaluation dataset and study how biases from the pre-trained models contribute to observed hallucinations. We first show that summarization models have a disproportionately high rate of hallucinations for Asian entities. We then propose an intrinsic mea- sure to understand how these ethnicity-specific hal- lucinations may arise from biases in the pre-trained …

Content truncated to 500 words